Step CI works programming-language independent and for different API paradigms (REST, GraphQL, XML).
Our CLI and test runner are available on GitHub (https://github.com/stepci) under the MPLv2 license.
Since our last launch, Step CI is now able to generate automated tests for your API based on your OpenAPI (Swagger) spec. This saves you a lot of time as you never have to write and maintain your tests again!
We would like to invite you to try our tool and give us feedback! Please star us on GitHub, if you like what we are working on!
We are very thankful for your attention and any feedback or suggestions we receive from you :)
Mish and Sebastian from Germany
The Step CI engine itself looks good though. It looks like a cleaner, but less powerful version of a tool (open source, build in-house) we used when I worked at OVHcloud, Venom: https://github.com/ovh/venom
Here's an example test file for the HTTP executor of Venom: https://github.com/ovh/venom/blob/master/tests/http.yml it's very close to Step CI format.
I'd still use Venom because it's way more powerful (you have DB executors for example, so after executing a POST request you can actually check in DB that you have what you expect) and I prefer focusing on actually writing integration tests instead of generating them.
Maybe this post sounds harsh (I feel it as I write it because I have strong feelings against test generation) but I think your approach is a good one for actually writing automated tests. Testing APIs declaratively like this has a great benefit: your tests work on an interface. You can migrate your API to a whole new stack and your tests remain the same. I did it multiple time at OVHcloud: one time migrating a huge API from a Go router to another (Gin->Echo), and another time migrating public APIs from a legacy, in-house Perl engine to a Go server.
> I prefer working with no tests than automated tests
We hear this often from our user-interviews and this is something we would like to change
I have just learned about Venom from a fellow hacker yesterday. We're more focused on Web APIs and offer more features there. The key difference is that we're not just a CLI tool, but also a library, that you can use in other applications.
Our end goal is to free you from writing tests
Thanks for your critical feedback.
Could you please tell us more about your strong opinion against test generation?
I don't believe generated tests can test anything relevant, and to me they're just a kind of mental load. One day they break, for who knows why and you have to fix them, not knowing how they work or what they do, even doubting the value they add.
To me tests should be business focused and thoroughly thought out. Knowing that your endpoint returns a JSON with a certain format is far from enough:
- it also has to return data that makes sense, generated tests cannot give me that
- and more importantly, the API endpoints that are the most useful to test have side effects. A PUT will probably modify data in a database somewhere. A POST may trigger an asynchronous action. Inputs and outputs are worth testing thoroughly, but you have to check the side effects too. In a good test you'll check what's in db before and after your actions. That's why I love Venom, because it allows me to precisely check the state of all my system.
To be fair, I also have strong feelings against generated code in general because it's always been an impediment more than anything. I worked with JHipster for some time (a Java generator) and it's still nightmare fuel to me.
Again, take this with a grain of salt, others may have a different vision on how testing should work.
Thanks for sharing Venom!
Certainly, there are lots of tools in the space, both SaaS and open-source. Mish and I can name you a few!
However, there’s one thing these tools seem to be missing - a greater goal.
Our vision is to give you a tool that will free you from writing and maintaining API tests ever again - in any shape or form. What you currently see from us is the first step!
The linked repo seems to have a greater goal.
Thanks for your question!
We allow you to capture content like headers, response and cookies between steps using captures, check out the "Captures" example in our Demo
Here's a documentation with examples for all capture types: https://github.com/stepci/stepci/blob/main/docs/workflow-syn...
also: we manage cookies for you automatically
Given the raise of CI/CD, I really think these kind of tools (CLI tests on HTTP requests), based on a simple format, will be really important. We've build Hurl (https://github.com/Orange-OpenSource/hurl), that shares a lot of similarities with Step CI (plain text instead of yaml, captures, jsonpath, xpath etc...). I will shamelessly take inspiration for some new features (like GraphQL for instance)!
Our Matcher functionality was actually inspired by Hurl (you call them predicates!)! We would take your inspiration from us as a huge compliment :)
The main difference between Hurl and Step CI is from our persepective an ideological one. What do you think?
Hurl is a CLI tool, while Step CI is a library. This means that you can include our Test Runner (https://github.com/stepci/runner) and use it with other tools. For example, you can combine it with testing frameworks like Mocha, Jest, and Ava (https://github.com/stepci/stepci/blob/main/docs/using-test-t...) or build your own platform on top of it.
That's what's so nice about open source.
Our OpenAPI (Swagger) integration is currently in Beta and only available through our website. We acknowledge, that it is not ready for prime-time just yet
Everything we do is open-source. You can see the list of issues in the repository and contribute if possible: https://github.com/stepci/plugin-openapi/issues
Disclaimer: We have tested our OpenAPI integration with the Petstore example and some other API definitions we had available. In both cases the generated tests would run after only some little adjustments
If you have any questions, feel free to post to our GitHub Discussions: https://github.com/stepci/stepci/discussions
You can generate tests from your OpenAPI definition using the “import from OpenAPI” button on our website.
The tests are generated from your request/response examples. If there are no examples given/available, we will generate examples for you based on the schema provided.
> "never have to write and maintain your tests again!"
this feels a bit misleading though.
What are you doing step ci? :D
Just give us a little more time and hopefully your support. :)
https://en.wikipedia.org/wiki/Synthetic_monitoring
You can subscribe to our newsletter if you want to get updates once we add new features or make new tools!
Great question. We are aware of Schemathesis
I believe the main difference is that our solution is very configurable, unopinionated and no-code. Schemathesis doesn't offer you an easy way to configure tests, relies on OpenAPI instead and for anything more complicated requires you to use Python
Maybe you could add something else?
I'd add that Schemathesis is essentially a fuzzer, where from the first glance Step CI is not (correct me if I am wrong).
We're glad you liked the idea! I had this idea in my head for about 2 years until I met Sebastian and we decided to give it a try!
We already support sending GraphQL queries and validating the responses
Soon we will also be able to generate automated API tests based on your GraphQL Schema. You can watch our main repository (https://github.com/stepci/stepci) or subscribe to our newsletter to follow our progress!
Thanks in advance for checking out our tool!
``` Uncaught YAMLParseError: The : indicator must be at most 1024 chars after the start of an implicit block mapping key at line 5551, column 3:
# $ref: '#/components/schemas/String'
schemas:
^^^^^^^Edit: Apparently I have some validation errors when pasting into the Swagger Editor. That must be why; will have to fix those up and try again.
We too like good UI! Our website is built with Nuxt and the Demo with CodeMirror 6