@ayufan had a great suggestion for us to spend a week on a singular focus, Memory Reduction, with a fast feedback loop. ## TL;DR; Pick a week for the whole team to have a synchronous focus and delivery of a Memory Reduction topic. ### What? What does this look like? In the pre-covid days, I might have suggested something that looks like a [Fast Boot](https://about.gitlab.com/handbook/engineering/fast-boot/) where we gain approval to meet and work synchronously as a team on a common goal. Given the current environment, we will have to do this remotely but highly synchronous. ### How #### Prior to the week Start creating **Topics** as a comments of this Epic. These topics will then be grouped by ~discoto. Once we pick a Topic, we would convert that into issue, and all relevant discussions would proceed within an issue. #### During the week "Start" each day with a synchronous discussion. The time for this synchronous discussion should be when all of the developers are available. Those of us in the US can catch up by reading notes, issues and watching the videos. Be sure to record these synchronous discussions. Treat each day like its own iteration. What can you accomplish within that day? Break it down, keep it small and be prepared to discuss your findings the next day. Gather all tools and created scripts as part of the https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week. #### The following Monday Sharing is caring. Be prepared to share what we have accomplished as a team. What impact did we have on the product? Did we make a meaningful impact on our memory footprint? What the outcome a detailed roadmap to have an impact on the memory footprint? Was this exercise not a good use of our time? All of these are good outcomes and we should be prepared to discuss and present our findings in our regular team meeting. Treat each day like its own iteration. What can you accomplish within that day? What can you ship, share or complete within that day. Break it down, keep it small and be prepared to discuss the next day. ### Why? There are a few benefits to trying this model. First, it aligns the team on a common and specific goal. We've been taking on 2-3 initiatives at a time and have not yet had an opportunity to have a singular focus as a team. Second, this could be a fantastic team building exercise. Third, it's always good to try new methodologies. We've been operating in a similar pattern since we became a fully formed team. By trying something new will give us a new perspective on whether we are a [highly performing team](https://gitlab.com/gl-retrospectives/memory-team/-/issues/28). This exercise will help to reinforce our strengths and expose us tom some ideas on how we can continue to improve as a team. Also, it's an opportunity to have a quick and meaningful impact on memory reduction within GitLab by shipping something awesome. <!-- start-discoto-summary --> ## Auto-Summary :robot: <details> <summary>Discoto Usage</summary> --- > **Points** > > Discussion points are declared by headings, list items, and single > lines that start with the text (case-insensitive) `point:`. For > example, the following are all valid points: > > * `#### POINT: This is a point` > * `* point: This is a point` > * `+ Point: This is a point` > * `- pOINT: This is a point` > * `point: This is a **point**` > > Note that any markdown used in the point text will also be propagated > into the topic summaries. > > **Topics** > > Topics can be stand-alone and contained within an issuable (epic, > issue, MR), or can be inline. > > Inline topics are defined by creating a new thread (discussion) > where the first line of the first comment is a heading that starts > with (case-insensitive) `topic:`. For example, the following are all > valid topics: > > * `# Topic: Inline discussion topic 1` > * `## TOPIC: **{+A Green, bolded topic+}**` > * `### tOpIc: Another topic` > > **Quick Actions** > > | Action | Description | > |-------------------------------|---------------------------------------------------------| > | `/discuss sub-topic TITLE` | Create an issue for a sub-topic. Does not work in epics | > | `/discuss link ISSUABLE-LINK` | Link an issuable as a child of this discussion | > --- </details> Last updated by [this job](https://gitlab.com/gitlab-org/secure/pocs/discoto-runner/-/jobs/894852331) * **TOPIC** Let's try using ~discoto in this to organize our discussion threads https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_439278061 * **TOPIC** Copy-on-write flagging in MRI https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_439791431 * Time intensive https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446918734 * **TOPIC** Investigate used gems https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446589283 * Split gems into groups like web and worker https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_448371505 * **TOPIC** Topic: Investigate used gems https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/5 * **TOPIC** Investigate memory content https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446589562 * **TOPIC** Topic: Investigate memory content https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/4 * Before forking Puma, only 130MB of heap are accounted for by Ruby https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/4#note_449223823 * We're holding 50MB of string data in memory https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/4#note_449767013 * What data do we have in memory? https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/4#note_451032843 * **TOPIC** Try to hack `spring` to run in `production` and `omnibus` https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446589848 * **TOPIC** Topic: Run Omnibus with Spring https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3 * enable spring support https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3#note_448787767 * significantly better start-up time https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3#note_448789439 * instantaneous `rais console` https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3#note_448789728 * GPT results 2GB without and with `spring` https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3#note_448792167 * Idle memory without and with `spring` https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3#note_448846402 * After GPT memory without and with `spring` https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3#note_448866574 * Idle memory after adding running `Puma single mode` https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3#note_449038225 * Runtime (after GPT) memory after adding running `Puma single mode` https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3#note_449086390 * Configuring `zswap` with `spring` and `Puma single` https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3#note_449183760 * Cross-link without `spring`, but with `puma single` https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3#note_449414549 * Alternative to spring https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3#note_452745361 * **TOPIC** Check the impact of swap https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446590183 * **TOPIC** Puma in single mode https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446590625 * **TOPIC** Research and measure running Puma in Single mode https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/2 * **TOPIC** Try to share `iseq` https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446591225 * **TOPIC** Try to load parts of application https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446592164 * **TOPIC** Try to run GitLab on JRuby https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446592506 * TruffleRuby could be another option https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446924387 * General thoughts on alternative runtimes https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_447962796 * Thoughts on JRuby https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_447962796 * Thoughts on TruffleRuby https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_447962796 * I think we should stick with MRI https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_447962796 * **TOPIC** Puma 5.0.x https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446592668 * **TOPIC** Check GitLab performance with Puma 5.1.x https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/6 * **TOPIC** Ruby 2.7 https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446593060 * **TOPIC** Investigate Ruby 2.7 GC compaction https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/1 * Aaron Patterson's talk on compaction https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/1#note_448192606 * How large is the Ruby heap vs OS heap? https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/1#note_448213147 * What is our heap layout before and after the Puma `master` starts https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/1#note_448344648 * Manually triggering GC https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/1#note_448374279 * What impact does compaction have on Puma procs https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/1#note_448768588 * What happens to shared memory after compaction? https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/1#note_451497380 * **TOPIC** Investigate a big difference between idle and runtime memory https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446593648 * **TOPIC** Measure the size of the features https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_446600612 * Could we use systemtap for this? https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_447191641 * **TOPIC** Disable unneeded services in Omnibus https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_447190273 * **TOPIC** Disable JIT https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_447971118 * **TOPIC** Gather baseline performance https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_448785282 * **TOPIC** Topic: Gather baseline performance https://gitlab.com/gitlab-org/memory-team/team-tasks/-/issues/75 * install & configure https://gitlab.com/gitlab-org/memory-team/team-tasks/-/issues/75#note_448193016 * GPT config https://gitlab.com/gitlab-org/memory-team/team-tasks/-/issues/75#note_448193993 * baseline results https://gitlab.com/gitlab-org/memory-team/team-tasks/-/issues/75#note_448778883 * **TOPIC** Figure out what holds memory https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_449288525 * **TOPIC** GC parameters https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_450669520 * **TOPIC** String deduplication https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_450687447 * **TOPIC** Remove `Gitlab::Metrics::Instrumenatation` https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_450701846 * **TOPIC** Remove `Prometheus` metrics https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_450702653 * **TOPIC** Tune GC params of all other Ruby processes https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_450707833 * **TOPIC** Discover unused code https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_452737874 * **TOPIC** Extend RubyVM to provide a detailed GC compacting details https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_452738077 * Done with: https://gitlab.com/gitlab-org/gitlab/-/issues/289842 https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_457144644 <!-- end-discoto-summary -->