UI is pretty but often breaks. Is unusable when you have a pipeline with dozens of concurrent jobs.
Concept of teams is fine, but when you have to switch between them, even with oauth authentication, it is a pain.
Job concurrency control is binary.
We never have been able to have worker pool scale down without a hiccup, always some darn worker hogging containers and atc not removing it, leading to stalled pipelines.
That overlay network it comes with, garden thingy, creates so many problems and solves just one...
Oh and not having BUILD_ vars available in tasks is rude, thank you very much, but there are cases when it is just mandatory and concourse makes it impossible to do.
At least new version has better secrets handling, previously it was a joke.
Find this hard to believe - we have some pretty spectacular pipelines which render without a hiccup.
Also, when you have around 100+ resources and jobs in one pipeline, in a vertical plane, they are impossible to identify
Can you expand upon this a little bit? Garden is the containerisation piece, what about it creates problems and what single problem do you think it is solving?
Are you running it on a potato? :) We have massive builds that have never had problems in the UI.
Admittedly I don't do it often, but I haven't experienced a problem with it when I have.
That said, the docs around implementing resources still needs some improvements. Whatever I learnt was from cloning and modifying existing resources.
A practical impact of this issue is that we cannot simply reprocess a previous deploy step, with the old artifacts and inputs in a middle of an outage.
My issues:
- everything has to be in a docker container (not all things can run in garden)
- Just trying to find logs is a pain, and getting the UI to show a full log output is basically impossible
- each install of concourse may need an entirely different version of fly
- There was no way of retriggering a CI run on a single PR (without force pushing to the branch, which removes Github reviews)
Jenkins + Jenkinsfiles + Pipelines provides all of the good bits of Concourse, with none of the Pivitol enforced workflow + tools.
I get why CloudFoundry uses Concourse - the build process is so arcane that you basically need to run the full CI locally, but I really do not get the hype for other projects.
Its currently impossible* to use Jenkins in a CD environment where you want to deploy by tagging your git project[1] and allow rollbacks because Jenkins doesn't treat tag's hashes (only the commits the tag points to).
Unfortunately, hack such a fix is not very pleasant either, because Java, and i don't think the Jenkins developers are interested in it either, at least not if just a few people ask for it.
It may look as a silly usecase, but i believe its one that mostly fits when you have a dynamic server farm (hosting on aws with autoscalling, for example) and don't want to auto-deploy on master commit.
[1]https://groups.google.com/d/msg/jenkinsci-users/mYxtDNMz1ZI/...
edit-> * aktually its not impossible, but pretty hacky and ugly, it would be nice to be able to rely on jenkins own tools for this.
When you want to work with CD in a particular way.
A simple fix to the above issue would be to have a "production" branch. When you want to do a release, you merge to the production branch. A roll back would just be a revert of the commits since the last merge to production, and could even be tagged.
If you are doing CD by allowing devs to tag random branches, cool, but I would not call it a "failure" on the Jenkins side.
> Unfortunately, hack such a fix is not very pleasant either, because Java, and i don't think the Jenkins developers are interested in it either, at least not if just a few people ask for it.
Its a webhook - that can be in any language, or if you use Git(hub|lab) a config item.
I keep running into this: we want to extend or change Jenkins behaviour, and since it's open-source, it's possible to get up-to-speed on the codebase & make that change. But getting it into a state which matches the mindset of the project maintainers is tough! Trying to convince someone that your use-case is not only permissible but is actually a good idea over ticket comments is not what we want to spend our time on.
A CI that runs one job (test) for lots of branches is differnet than a multi-pipeline CI that runs lots of jobs for a single branch (which is what concourse aims to be).
(not associated with buildkite, just a very happy customer)
Can you provide an example of what you'd want to do but you aren't able to?
Have you tried `fly watch`? It will stream logs from in-flight builds, or show the full output of completed ones.
Or - I can go to my Jenkins build page, and click "Show full log".
For me, I find the initial quick searching through logs much quicker in a browser.
And, I have found that on longer builds the web UIs log streaming can actually crash the browser.
What? It's just STDOUT.. Are your logs going to a file or something?
Different stages go to different log files, so you don't end up with a 10k log file.
- Truly a pipeline based stages. No messing around with git branches/tags for each environment release.
- Ability to combine multiple builds together.
- Build dependencies. Ability to trigger project-B build when project-A build succeeds. (Docker Cloud Builds have something like this.)
- Use the same artifact across multiple build stages.
- An option to promote a build to the next step either manually or automatically.
Amazon's internal Apollo system was amazing. AWS Code Pipelines kind of does some of these things but it's every limited and hard to work with.
> - Truly a pipeline based stages. No messing around with git branches/tags for each environment release.
That's how we use it. We have two or more stages for each environment (repo upload + installation, often tests) that are simply chained.
> - Ability to combine multiple builds together.
You can do fan-in and fan-out between pipelines (you can combine multiple pipelines in a DAG).
> - Build dependencies. Ability to trigger project-B build when project-A build succeeds. (Docker Cloud Builds have something like this.)
As above, with a graph of pipelines. We use this to trigger rebuild of some projects when libraries change.
> - Use the same artifact across multiple build stages.
You can reference and retrieve artifacts from all previous stages, even stages from upstream pipelines. The syntax for referencing takes some time to get used to.
> - An option to promote a build to the next step either manually or automatically.
automatic is default, but you can have manual approval steps, optionally with permissions attached to them.
Plug: I wrote a book that uses GoCD for its examples: https://leanpub.com/deploy
It's a hybrid hosted model -you supply your own workers (they provide a CFN template to make that part easy), everything else (e.g. web UI/API) is hosted
Within a pipeline, you can 'pass' an output of one step as a triggering input to another. Across pipelines, you can use e.g. an S3 resource to save state; the job to be triggered looks at that S3 resource and triggers when it changes.
> Use the same artifact across multiple build stages.
Yep - you can pass the output of one job to be used as the input of another.
> Ability to combine multiple builds together.
Yep - have a look for "fan-in" in the docs.
> An option to promote a build to the next step either manually or automatically.
You can manually trigger steps from the UI; you could also have a resource that triggers a job automatically on some condition.
There was talk of Python + Linux support a few years back, a shame that didn't take off.
You can do Linux today actually - you can connect to machines over SSH, run bash scripts, etc. Today there's a Mono dependency, but in the next couple of weeks we'll be rolling out a beta that removes that dependency.
Python scripting support is on the roadmap - :-)
The UI is clunky.
The abstraction layer is too low and leads to a lot of repeated YAML. Which leads to YAML programming.
There are simple scenarios like deployment rollbacks who are hard to do.
For some reason they decided to develop their own container engine which leads to all sorts of trouble and maintenance issue. It's generally slow and we had 100% CPU usage when the worker was doing almost nothing.
I have used for 4 months and it was only problems. Gitlab CI is much better. Or even Jenkins is better.
I've used teamcity, jenkins, gocd, circleci, concourse, and travisci. For multi-project systems, concourse is king. (I like travisci for by-itself, non-system projects)
> The UI is clunky.
What? You just said the UI looks good... It's simple and clean; everything is async javascript (no page loads).
> The abstraction layer is too low and leads to a lot of repeated YAML. Which leads to YAML programming.
Which is an intentional choice. If you don't like YAML, use one of the MANY yaml abstraction layers of your choice...
> There are simple scenarios like deployment rollbacks who are hard to do.
First of all, a CI system shouldn't be your answer to rollbacks. Your deployment system should handle that. Secondly, assuming your deployment system can do rollbacks, concourse has on-fail jobs that can trigger rollbacks just fine.
> For some reason they decided to develop their own container engine which leads to all sorts of trouble and maintenance issue. It's generally slow and we had 100% CPU usage when the worker was doing almost nothing.
garden is used because cloudfoundry builds it. It is not slow... it is a light layer on top of runc (as opposed to docker which is a rather heavy layer on top of runc). You should pretty much never have to care about it, and in 3 years of using concourse I haven't had to - and we have some pretty gnarly large pipelines.
Also I call rubbish on your 100% CPU. I have two workers t2.xlarge workers running 22 and 30 containers (like, right NOW) and neither is above 10% CPU (which, actually, I should make those a lot smaller). Don't run workers on a potato and you'll be fine.
> Gitlab CI is much better. Or even Jenkins is better.
Ohhh you're trolling.
I meant that when just looking at the website, Concourse looks good in terms of what it has to offer. For example it offers proper build pipelines, something that is lacking in most other CIs.
Maybe it's clean to you, I found that our developers where generally confused by how the releases were working.
> Which is an intentional choice. If you don't like YAML, use one of the MANY yaml abstraction layers of your choice...
I don't mind YAML too much but adding a templating language on top is generally clunky. Gitlab CI also uses YAML, can also build pipelines and doesn't require to generate the YAML.
> Also I call rubbish on your 100% CPU.
It might have been due to a misconfiguration (we were using c4.4xlarge). I bet that you have deployed Concourse using BOSH, the only truly tested deployment method.
I was on CF Buildpacks for a while, which is one of the earliest and more intensive users of Concourse.
Earlier versions of Concourse did stumble upon fairly hairy btrfs bugs that would enthusiastically choke CPUs to death if you had more than about 100 containers on a worker. This was particularly bad if, for any reason, you had a lot of jobs launching at once (I'm looking at you, 6-releases-simultaneously-NodeJS).
I can't remember if it was fixed by a kernel upgrade or whether they ditched btrfs. Either way it got fixed and I haven't seen it since.
Disclosure: I work for Pivotal.
The UI was originally designed with one end in mind: "Show me what is broken and take me there as fast as possible".
I and others have given lots of feedback about the strengths and limitations of that orientation, particularly trading off between a job-oriented and resource-version-oriented view of the world.
> There are simple scenarios like deployment rollbacks who are hard to do.
I've had this with the CF resource. The Concourse-y solution would be an undeploy resource.
> For some reason they decided to develop their own container engine which leads to all sorts of trouble and maintenance issue.
The container engine is garden-runc, the actual containerising is the same code as Docker's. What I like a lot less is container orchestration. I have complained about this in a ha-ha-only-serious matter.
Disclosure: I work at Pivotal.
All things will break horribly if the conditions are right. It's unreasonable to assume that the things which work in [insert your current CI] will work in Concourse. It's still a new and relatively immature product, but it works well in most cases.
Half the secret to a good Concourse experience is not upgrading it in-place - stand up fresh deployments. The other half is gradually transitioning between Concourse deployments, because bad versions have been and will continue to be released - mistakes are only human. As long as you share the Concourse vision and are willing to keep up with the pace of change - not everyone can or wants to - then it's an amazing CI.
Concourse still makes me excited, even after many years of hard lessons, because it is a genuinely innovative approach to building better software. Most miss this, and I understand why, but give it time - the ideas behind it will mature and become the norm.
Even though Concourse can work really well, it's not always the best choice. Make it better if you can & want to, use something else if it's easier. There is no right or wrong, just preferences : )
Concourse is difficult to come to from other CI tools. A little more aloof. There have been real, serious implementation difficulties.
But I kinda love it, because it's a handful of simple ideas that unlock incredible power. It goes beyond "build and test each commit" to becoming a full project automation tool, a software manufacturing robot. When I talk to people about Concourse, I tell them: your pipeline and your tasks are production code. Keep the discipline, care and engineering practices that you bring to the apps and services you create.
The problem is that most of the deep tribal knowledge about how to get started and how to best apply it is locked up inside a handful of organisations, most notably Pivotal. I had been working on a video series which was meant to walk through both the concrete business of building pipelines, as well as the concepts of how best to do so.
Unfortunately visa conditions got in the way and I have abandoned that effort. Interested persons are welcome to email me for links to the first 4 episodes that I made, but be aware that it stops even more suddenly than Firefly.
Disclosure: I work for Pivotal.
Those containers can run the Docker daemon though, if you like.
Archlinux for example does not compile in user namespaces. Vagrant is the most cross-platform option for quick local concourse testing.
The main criticism I saw on Jenkins and Go.CD vs. Concourse was that Jenkins Pipelines aren't first class and that it's easier to export configuration(in that regard Concourse > Go.CD > Jenkins). On the other hand Jenkins and Go.CD supports extensions, which Concourse touts as a feature.
I also want the CI builds to create my base boxes with packer in multiple steps. And I somewhat want to be able to hand over the stuff to ops at some point to be able just stay alive for the next 5 years or more. Would anyone know if it makes sense to even consider concourse or go.cd or some other CI/CD solution and if so which?
Obviously the boxes need to be used as artifacts and everything has be on premise as well.
Disclosure: I work on Jenkins.
Admittedly I ditched jenkins before pipelines became a full feature, but my understanding of them in jenkins was that they are mostly for scheduling of jobs. IE, run job A after job B
Concourse passes inputs, outputs and resources between jobs as the ONLY STATE, and jobs trigger based off of changes on resources or the availability of new inputs.
I think that when you sit down and look at the two products, jenkins is great at running scripts, and concourse is great at managing code versions.
In the end we actually run both concouse + a script runner; this allows concourse to manage the tagging, builds, releases, and testing; but still allows us to run ad hoc scripts that concourse doesn't do well.
I'd be interested in taking another look at jenkins now that pipelines and the groovy DSL is solidified, but I get the idea that they still fill slightly different needs
This is all in-line with the "everything is a plugin" methodology, but in my mind that way of thinking is one of the biggest hindrances to Jenkins, second only to the lack of a real database powering it.
Overall, I would choose Jenkins first unless you know there's something GoCD can handle significantly better.
This is certainly possible using concourse, I've done it myself on a few teams. We had one job that started with a base ISO and used the virtualbox-iso builder to apply updates and build an OVA. Then a second job would trigger whenever a new OVA was built and used the virtualbox-ovf to apply a different set of provisioners. Since the virtualbox-ovf builder uses an OVA/OVF file as both it's input and output, you can do that as many times as you'd like.
This is rather annoying if you want to run a copy on your local network say at home. Its very frustrating because the fly command solves the biggest issue with IWOMM (it works on my machine) by allowing you to run code and tests on another machine before committing anything.
I think from memory I tried using self signed certs and this also had issues for one reason or another.
That said it is still the best CI system I have used to date.
Nit: Not familiar with Concourse, but this is a design decision. Go stdlib let's you bypass SSL/TLS verification.
For example, I have actually implemented patterns within Jenkins to force all jobs to run inside of containers. A job is just a container + command with some linking using the jenkins pipeline plugin that reads some json configuration to determine how jobs are linked.
The primary issues I have surround the fact that my company uses kubernetes, thus we have no insight into the runc containers. Load balancing in concourse is non-existent. If a worker goes down due to load, if you bring it back up, it's going to go down immediately from all the jobs that have been triggered while the worker was offline. Not only that, the resource requirements seem pretty high. Recently a concourse worker stalled because the amount of volumes/images it was caching was over 100 gigs, and not knowing the internals, I wasn't sure what the best way was to clear this cache. Having to tell the infrastructure team that we just need to spend more money is a hard sell when we've upped the cpu, memory, storage, postgres disk, all more than once. I understand that different images have vastly different sizes, and jobs different amounts of work, but their need to be some clear suggestions for sizing. If there exist them now, I apologize but I haven't seen them.
So yeah I've had some fun developing in it, but more help making it reliable would be really nice. Also if kubernetes is absolutely the wrong way to run it, which it seems like, I'd have to be provided a better/easier alternative to really become an advocate.
Final note: has anyone actually setup metrics/monitoring for concourse that doesn't know BOSH? The docs describing it seem huge unless you already have the infrastructure pieces. Let's setup riemann (we already have statsd and no experience with riemann), emit to influxDB (we have prometheus, no experience with influxDB), then use Grafana (ok we already have that). I just wanted a better idea of disk, cpu, mem, # of containers, lifecycle without having to setup all these new pieces of infrastructure. Finally, just not that interested in BOSH, which all of the example metrics repo's are.
Perhaps ironically, we ended up doing some development around the edges of the CI product we ultimately selected (GitLab).
So I think the feature we'd want to make that easier is an actual global code search API!
I wrote an issue for it here: https://gitlab.com/gitlab-org/gitlab-ce/issues/29347
This is actually a brilliant insight. I'd never thought of it that way. And I'm a one-eyed raving Concourse fan.
The best CI is the one where you write code to build/deploy etc.
Wrangling with some moronic UI exposed by the CI product to abstract code is simply wrong. Don't do CI/CD like this.
That lets teams share Concourse task and resource definitions. Many teams (including mine) publish our pipeline definitions too [0]. This enables "tooling" type teams build components that get a lot of re-use by other teams.
disclaimer: I work for Pivotal.
[0] e.g. code.cloudfoundry.org/cf-networking-ci
But I think there's a difference between a mostly complete CI product with API extension points, and a "batteries not included" CI toolbox.
Our team's been using it since it's initial releases. It's been nothing short of disastrous for all but the smallest pipelines.
The design is great. Keeping configs in yaml instead of little white boxes in a Jenkins database is much better. Pipelines as a first class concept. It feels inspired by a functional programming language. You get great build reproducibility since there are no workers that get dirtier over time if you forget to clean up. The resource model is awesome. Very cool stuff. I'm hoping every CI system learns from what's here. Second to none in design.
However it performs like a dud. No scheduling to speak of, just runs everything as soon as it can. We've run into nodes dying under load (-not- underprovisioned, could run all these jobs manually at once on these monsters). We've run into problems with volume reaping, fork bombs, ui freezes, everything under the sun.
I really like Concourse and will hopefully one day be able to come back to it when its implementation is as solid as its paradigms are.
But I'd avoid use for now.
I've used concourse as a consumer for 3 years and I've very, very rarely seen any of the problems you're describing, even on the older versions and certainly not in the last year or so.
> no scheduling to speak of
Concourse has a massive scheduling system built into it.. https://github.com/concourse/atc
Furthermore, you can configure jobs to run in serial (default is parallel).
> ui freezes
Put your `web` binary on a decently-sized VM and your problem should disappear. Also, don't have your workers on the same VM as your `web`.
ATC is a more of a dependency scheduler. The code [1] shows that it basically gets all pending jobs, and then runs them. There's no concept of queueing or maximum number of jobs, you just have to hope your limits are high enough (max containers, max tasks in systemd, max fds in the same) and that your machine doesn't fall over in the attempts.
The "massive" scheduling system also has no idea what nodes need work and which do not [2] so the idea is to heavily overprovision until it doesn't fall over (on top of already beefy requirements which others have alluded to in this thread).
You can not serialize multiple pipelines. Only within pipelines. If I have 10 pipelines, they will all run independently and there's nothing you can do about it other than attempt serialization with the pool resource (which we've recently had problems with - it also appears to be buggy and we're looking at submitting patches).
I've got a tremendous amount of experience with this system and I believe it's everything I made it out to be. The rebuttals you've provided to my issues are simply a lack of understanding of our context, not every user will have the same experience with any given product.
[1] https://github.com/concourse/atc/blob/master/scheduler/sched...
Concourse also isn't really made to work well with the Git Flow, there is no builtin way to run the CI on multiple branches (there's a git-multibranch community resource which requires redis at some point). we're basically thinking about changing our workflow to trunk-based, but it still feels weird to me that we might change our workflow to fit our CI better.
that being said, I personally still really like concourse and it's fun to work with.
Refused to execute script from 'http://127.0.0.1:8080/public/index.js?id=932c553753eec4ef375... because its MIME type ('text/plain') is not executable, and strict MIME type checking is enabled.
Pros - Setup was really easy.(I didn't use bosh). - Both server and build pipeline configuration (Yaml file) are easy to backup and recreate.
Cons - When I evaluated it , it had a few bugs which forced me to restart the server almost daily to keep it working. It should be more stable now.
-
How can I feed this to Jenkins and plot it over time?
https://github.com/EngineerBetter/concourse-up
It'll give you a working, upgradable, auto-healing Concourse in one command.
They are pre-compiled go binaries, and require nothing other than a modern kernel! If you want an example you can see an example in concourse's upstream docker config
Rather than "Rather than a myriad of checkboxes..." it should be "Rather than myriad checkboxes..."
> Usage Note: Throughout most of its history in English myriad was used as a noun, as in a myriad of reasons. In the 1800s, it began to be used in poetry as an adjective, as in myriad dreams. Both usages in English are acceptable, as in Samuel Taylor Coleridge's "Myriad myriads of lives." This poetic, adjectival use became so well entrenched generally that many people came to consider it as the only correct use. In fact, however, both uses are acceptable today.
