Knowing when a bug fix is live

Hi,

I know I have asked this question before however I was not able to solve the issue with past responses and I think there could be some improvement with this process. Perhaps someone can suggest an easier way to know when bugs have actually been pushed to production. I have tried alot of things, nothing super straightforward.

Question:
When a bug is marked fixed and I see it has been merged to main, How do I know when the fix is actually live and has been pushed to production?

Previous answer, please correct me if I got any of this wrong:
All fixed bugs and features should appear in a milestone. Milestones are deployed monthly. You should be able to know if a bug is actually live by tracking it to it’s milestone and keeping track of when a milestone is deployed.

Challenges with the current process:

  1. It can be quite difficult to track a bug to it’s milestone. In fact often bugs are often resolved and closed through a series of interrelated tickets, making it almost impossible or not worth the time to track a fixed bug to it’s milestone.
  2. It is not entirely clear to me when a milestone is actually pushed to production. Do I need to routinely check the milestone around the deadline date to check?
  3. There are a number of bugs that don’t make it into a milestone, although due to bugs being closed through a series of interrelated tickets, it is really hard to tell which bugs are actually not part of a milestone. How would I know when one of these bugs has been pushed to production?
  4. If a bug is merged to the main branch it is not an indication that the fix has been pushed to production, however evetually over time it is. Could some automated use of labels on the issue ticket indicate when a bug is actually live and can be verified in the wild?

I ask these questions with the healty intention of wanting to help verify bug fixes.
Previously I would try to track the bugs to their milestones with the intention of verrifying them as fixed but the time involved (sometimes 20 minutes to hit a dead end) was just not worth it.

Thank you and keep up the awesome work! I really believe in your prduct and your team Gitpod!

:

2 Likes

Hi @Devz3n! :wave:

Thank you for making an excellent point: It’s currently much too hard to know which fix or new feature gets deployed to production and when (even for core team members).

Also, many thanks for spending 20 minutes of your time tracking down bugfixes… hopefully we can make this task much easier for you!! :see_no_evil:

I’ll try to answer in two parts:

When is Gitpod deployed?

Currently, we aim for monthly release cycles (we do that since January 2021), where we implement features & bugfixes for a few weeks, and then deploy the latest main during the last week of each calendar month. (That week typically starts with a staging deployment of main on Monday, followed by extensive testing / fixing / polishing, and finally a deployment to production, typically around Thursday +/- 1 day.)

Caveat 1: We occasionally do additional “hotfix” deployments, on pretty much any day of the month. For example, if the regular production deployment reveals a serious problem, we may implement a quick mitigation and then follow-up by deploying a proper fix a few days later. Or, whenever an incident happens, or if a security vulnerability is disclosed, we can make a “hotfix” deployment where we selectively deploy only one or a few relevant commits (leaving all other non-urgent commits currently on main for the next “regular” deployment). FYI, you can detect most deployments because they change Gitpod’s production version tag and/or Gitpod’s workspace cluster generation (e.g. ws-eu08 in workspace URLs becoming ws-eu09), but unfortunately this won’t help you determine which commits are deployed or not.

Caveat 2: We’re also constantly improving the deployment process, and we’re currently looking for ways to deploy to production more frequently (ideally all the time, continuously, with automated tests and staged deployments to prevent any bugs from affecting users) and more selectively (we would love to deploy only the components that received changes, e.g. there is no need to cycle all the dashboard front-end servers when deploying a fix to the workspace image-builder – we already do that manually in some “hotfix” deployments, but would like to automate it). This isn’t finalized yet (as of June 2021), but our end goal is to reduce the time between a merged fix and its production deployment as much as possible.

Summary: All Gitpod commits are deployed to production by the end of each calendar month, but some commits can be deployed earlier than that.

How to know when a specific change is going live?

From the above, the general rule of thumb is that any merged commit will be live on gitpod.io by the end of the month it was merged in, via the regular end-of-month deployment (or sooner if the change is considered “urgent”, like a critical bug fix).

There are a few clues that can help you specifically track down whether a change is deployed or not, but if you’re ever in doubt, please always feel free to ask in the GitHub issue whether the fix is live yet. (At least the commit authors should have a pretty good idea of when their changes are going live.)

Here are some sources of information I like to use, with varying levels of detail:

  1. Commits on the main branch – this can be quite overwhelming, but it’s the most complete and accurate source of truth regarding what’s going into production, because we deploy main at least once every month.

  2. The CHANGELOG.md file – we’ve recently started updating it again, and we ask all developers to add a brief summary line for each significant change made to Gitpod. If done correctly, this gives you a one-line summary for changes that may otherwise comprise multiple commits, issues and Pull Requests. Lines are added per month, so everything you see listed under “June 2021” is what we expect to deploy at the end of this month (probably around June 30th).

  3. The monthly milestones – as you’ve noticed, this gives a somewhat incomplete view, because not every change is added to these milestones. We use this more like a project management tool, where we collect issues we want to fix in the next cycle, then we do a bit of triage/prioritization, and then we work towards closing all issues from the current milestone (nowadays we assign a few developers specifically to fix issues in the current milestone, in a rotation that changes every month).

  4. https://www.gitpod.io/changelog – this is also somewhat new. We try to update this page after each regular deployment (i.e. at the end of each month) with a curated list of a few Gitpod changes that are important or worth highlighting, along with a clear and interesting description, and some exciting visuals when possible (e.g. a screenshot, GIF, or a visual representation of how a new feature works).

If any if this doesn’t quite make sense, or if you have any feedback to share, please do. We strive to make Gitpod’s development more transparent and welcoming, and likely still have a lot to learn along the way, so any hint or suggestion is very much appreciated. :pray:


An additional thought that came to mind (and maybe a more direct way to answer your question):

We used to have an issue label called “fixed-not-deployed” that we would add to issues where a fix has been merged but not deployed yet. Then, after each regular deployment, we used to go back through these issues to remove the label and close it if needed. Would this be something you’d like to see again?

3 Likes

Thank @jan for the detailed answer :pray:

I was going to suggest a label technique for each issue like you mentioned at the end of your answer but I thought that it would be too much work for the core team to keep track of and too prone to human error. If the label technique could be automated somehow that would work really well.

In the meantime how feasible is it to simply mark each merge into main with something like "in-dev-qa" and then remove it when that PR has been deployed to production?
One can track an issue to a merged PR pretty easily most of the time so labeling PR’s merged to main but not deployed to production could really help the with transparency internally and externally.

1 Like