Enable `dynamic_image_resizing_owner` FF via production console for all GL employees
Production Change
Change Summary
We want to run the Dynamic Image Resizing experiment.
To do that, we need to enable the Feature Flag (`dynamic_image_resizing_owner) for GL employees.
We run a smaller test already #2675 (closed).
It will only affect Gitlab Employees' avatars which would be resized in our service before being served. Content images are not affected. We expect there will be no difference in User Experience. We want to monitor the Workhorse performance, stability, and resource utilization.
Change Details
- Services Impacted - GL Workhorse
- Change Technician - @ayufan
- Change Criticality - C3
- Change Type - changeunscheduled
- Change Reviewer - @alipniagov
- Due Date - 15 Sep, 2020, 12:00 UTC
- Time tracking - 275 minutes
- Downtime Component - No
Detailed steps for the change
Pre-Change Steps - steps to be completed before the execution of the change
Estimated Time to Complete (mins) - 15
-
Start a sync call with the Memory Team members: Aleksei Lipniagov, Nikola Milojevic, Matthias Kappler, Kamil (if available), and make sure that everyone is on board. The ideal start time would be somewhere in between of 9:00 GMT and 13:00 GMT, but we could discuss it. -
Post the announcement to Slack:
Hi All :wave:
:speaker: The Memory team is going to test the Dynamic Image Resizing feature on production during the next hours. It will only affect Gitlab Employees' avatars which would be resized in our service before being served. Content images are not affected. We expect there will be no difference in User Experience.
We enable it today to monitor the Workhorse performance, stability, and resource utilization.
More details can be found at https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2692 and the epic: https://gitlab.com/groups/gitlab-org/-/epics/3822.
Please communicate in #g_memory or in the https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2692 if you have any concerns or you encounter any strange user avatars issues. :speaker:
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 10
-
Run:
[ gprd ] production>
usernames = ["sytses", "dzaporozhets", "marin", "razer6", "dblessing", "axil", "vsizov", "DouweM", "haynes", "rspeicher", "ayufan", "stanhu", "yorickpeterse", "rdavila", "grzesiek", "dbalexandre", "brodock", "tmaczukin", "rymai", "jameslopez", "zj-gitlab", "annabeldunstone", "felipe_artur", "twk3", "smcgivern", "jacobvosmaer-gitlab", "chriscool", "marcia", "alejandro", "tauriedavis", "nick.thomas", "eliran.mesika", "ahanselka", "ahmadsherif", "abuango", "arihantar", "xyzzy", "dimitrieh", "markglenfletcher", "mikegreiling", "pedroms", "WarheadsSE", "ClemMakesApps", "lbot", "ibaum", "athar", "cindy", "jivanvl", "balasankarc", "harishsr", "matteeyah", "kushalpandya", "toon", "collen", "jarka", "joshlambert", "bjk-gitlab", "ja-me-sk", "andrewn", "pmachle", "MadLittleMods", "reprazent", "amulvany", "xiaogang_gitlab", "blackst0ne", "mkozono", "mbell", "adamjohnson", "teemo", "timzallmann", "mrogge", "brittanyr", "simonwill", "Cbrady", "dosuken123", "dt", "vehernandez", "jramsay", "nolith", "ckatanda", "jarv", "jordan_goodwin", "jwoods06", "jbandur", "mbergeron", "dwells1", "NicoOchoa", "astrachan", "davinwalker", "csotomango", "jkrooswyk", "edjdev", "ccornacchia", "Larry-Gitlab", "fjsanpedro", "williamchia", "abuerer", "mayra-cabrera", "brendan", "jprovaznik", "gronk", "jimt2061", "digitalmoksha", "jrileyinva", "psimyn", "jkalimon", "chloe", "PaulDuffy", "Haydn", "Emily", "luke", "julie_manalo", "erica", "jdrumtra", "Elsje", "kyla", "evhoffmann", "rebecca", "sarahj", "PTMJR", "aolson", "pauldalmeida", "bcaldwell-gitlab", "jburton", "gitlab-bot", "tnir", "mhamilton", "hschuler", "bwilliamson", "Adam.Pestreich", "tpresa", "justinhaley", "disastor", "katokpara", "Cpepper", "rkompare", "k33g", "dennis", "abrandl", "pharlan", "jritchey", "andr3", "lyle", "dstanley", "atflowers", "cdmaurer13", "videmsky", "jacopo-beschi", "jmiklos", "tayloramurphy", "mdelaossa", "DylanGriffith", "plafoucriere", "gonzoyumo", "fcatteau", "kuthiala", "mikemiranda", "suripatel", "atanayno", "vgoetz1", "jeremy", "leipert", "jbushong", "kristie.thomas", "aoetama", "wwright", "annapiaseczna", "amimmo", "dsumenkovic", "johnjeremiah", "meks", "rostrander", "Mowry", "borivoje", "msnow1", "mrjkc", "LisavandeKooij", "deandre", "dangordon", "Apshegodskaya", "matejlatin", "asaba", "ashmckenzie", "iroussos", "danielgruesso", "dawsmith", "jareko", "brianwald", "mbeadle", "pslaughter", "lulalala", "ifarkas", "tcooney", "mmcb", "mjwalsh", "samdbeckham", "awinata", "wayne_zhao", "wvandenberg", "sbillow", "aciciu", "Alexissh", "leopardm", "cblake", "Jewell", "andyvolpe", "devin", "wortschi", "glopezfernandez", "cynthia", "shanerice", "ddavison", "KalebHill", "Tompsett", "rpaik", "jdavidson124", "jnguyen28", "Steevo", "erushton", "MarcKriz", "rvzon", "skarbek", "tristan", "mlapierre", "jyavorska", "praumann", "tkuah", "dfishis1", "mhuseinbasic", "dtacheny", "jurbanc", "steveazz", "pkunkli", "tatkins", "craig", "jerome", "bjung", "wchandler", "sliaquat", "rmarshall", "at.ramya", "emilyluehrs", "lmcandrew", "cody", "jgragnola", "jcolyer", "blunceford", "eread", "zkovacs", "rnienaber", "dplanella", "togor", "Finotto", "dappelt", "adietrich", "splattael", "Jennychapman", "ameliabauerly", "ShawnWinters", "jerasmus", "amandarueda", "dcoy", "ssichak", "Kellymurdock", "cheriholmes", "acarella", "gitlab-securitybot", "estrike", "rpereira2", "Kikeadio", "khair1", "AkeelaLogan", "tbarr", "A_Sharma", "RyanKimball12", "jhampton", "hphilipps", "shampton", "T4cC0re", "anazir", "mnguyen4", "k.mckinney", "rayana", "awaters3", "dhong", "davistye", "kencjohnston", "ecook1", "blabuschagne", "brytannia", "gtsiolis", "tristan.read", "engwan", "jvaleriano", "Joemckairnes", "pursultani", "ntepluhina", "ronell", "derek-knox", "Phuynh", "lwbrown", "nfriend", "ebrinkman", "farias-gl", "johncoghlan", "rajatgitlab", "sravindran", "jalex1", "darbyfrey", "DevOpsPhil", "cupini", "AnthonySandoval", "theoretick", "Ahoffbauer", "pharrison", "shaynes13", "proglottis", "jburrows001", "Alexand", "blazzarin", "jdsalaro", "jemcgrew", ".luke", "timnoah", "dperic", "syasonik", "tomi", "rkohnke", "mlebeau", "francispotter", "jrector", "nsarosy", "nlarue", "zeffmorgan", "patrickbajao", "mfluharty", "lmcnally1", "vshushlin", "afontaine", "scull", "pokstad1", "Haileyrae", "cteskey", "sreed1", "monaelliott", "amrvaljevich", "mpower", "ekigbo", "dmishunov", "igor.drozdov", "clefelhocz1", "mwasilewski-gitlab", "tigerwnz", "kevinchasse", "hugoazevedo", "mayanktahil", "mallen001", "bchurch", "melsmo", "mlindsay", "cmcfarland", "mksionek", "clenneville", "pshutsin", "markrian", "acroitor", "sahbabou", "akgoodwin", "cferreira1", "donaldcook", "weimeng", "sheetaljain", "ewegscheider", "rdouglas-gitlab", "cnovello", "Lconway", "cgudgenov", "vladbudica", "cbuchanan", "gitlab-release-tools-bot", "sarahghp", "tpazitny", "heather", "mloveless", "rossfuhrman", "dmor", "marcusbriancarter", "lisvinueza", "atiseo", "ealcantara", "igroenewegenmackintosh", "crollison", "sdaily", "jeffersonj", "Plumadore", "uhlexsis", "kmann", "rcobb", "mmaradiaga", "jbroullon", "PaulOakley", "llamb", "greg", "agroleau", "twoodham", "gitlab-it-opsbot", "Riyer0019", "kavas", "rdickenson", "kstithem", "ccasella", "amberlammers", "iganbaruch", "cfletcher1", "bmarnane", "broyer1", "drewcimino", "mvanremmerden", "bdowney", "ssilmon", "eggshell", "adesai1", "jagood", "cwalsh11", "cablett", "vkarnes", "mnichols1", "mbenza", "xanf", "joshikripa", "vrana3", "pgascouvaillancourt", "cmiskell", "krasio", "sdelea", "Tknudsen", "kyletsmith", "mterhar", "yofeldstein", "cdu1", "mpyle", "amberlammers-admin", "keithsnape-admin", "dsakamoto", "cwilliams3", "kmcknight", "dchevalier2", "tlavi", "kerrizor", "trizzi", "skamani", "sarahwaldner", "viktomas", "ctimberlake1", "spouliquen1", "bgal", "vsilverthorne", "Chris.loudon", "kmatharu", "EWhelchel", "abelardi", "mwilkins", "fabiopitino", "mendeni", "jbrennan1", "rwilson3", "craigf", "dcroft", "cjdewit", "skassabian", "NicoleSchwartz", "toupeira", "AnnetteK", "afonseca2", "cvillanueva1", "cmachado1", "fzimmer", "ghickey", "jennyt", "kxkue", "Kelsey.steyn", "lauraMon", "msmiley", "pprokic", "pfdistef85_git", "SaschaScholing", "sfwgitlab", "vstoianovici", "jbernstein", "phikai", "philbz", "dcsomers", "romerg", "alexkalderimis", "Gzaparkanszky", "marsjajones", "acooke", "dparry", "mtaft", "rbartholomew", "bbodenmiller", "egrieff", "nnelson", "imahros", "c-admin", "darawarde", "sming-gitlab", "TinaS", "asubramanian1", "rverschoor", "eslatt", "jeldergl", "chipdigi", "jboyson", "kaleyjohnson", "krogel", "bma", "ericrosenberg88", "sabrams", "dbodicherla", "anthonyot", "avielle", "craig-gomes", "cwillis", "mnohr", "mpeychet_", "msmith6", "jgendrano", "manojmj", "glen_miller", "jennykline", "kevans1", "jackib", "cmaxim", "chriscruz", "caneldem", "jmarquez2", "reneho", "hfyngvason", "akohlbecker", "qzhaogitlab", "nwoods1", "loriewhitaker", "alipniagov", "echin", "deuley", "dsatcher", "dpisek", "aakriti.gupta", "vitallium", "grantyoung", "Matt.Petrovick", "christiaanconover", "tyleramos", "j.carey", "Swethakashyap", "bmatturro", "crgraham87", "HaseebK", "tstadelhofer", "cnightingale", "ogolowinski", "ahegyi", "pranavpolisetty", "SharifATL", "sverdin", "cnwachukwu", "jennielouie", "melissar-admin", "ragimab", "mmacfarlane", "jblanco2", "iweeks", "marcel.amirault", "ebaque", "rmawhinney", "marie_curie", "ankelly", "steve_xu", "csaah", "jstava", "traci", "georgekoltsov", "ECorring", "dmurph", "tonyfeldman", "PhilEncarnacion", "vbrodsky", "jeanduplessis", "hfrederiks", "eg0nzalez", "PegSheridan", "nicki_peric", "mjbatgitlab", "codeEmitter", "dustinmm80", "Vatalidis", "Anja_L", "aalakkad", "floudet", "ggelatti", "SteveTerhar", "Tlauver", "xlgmokha", "dsharris", "rhyry1", "GitalyBot", "gitlab-crowdin-bot", "tom.houston", "rnalen", "abrandenburg", "icamacho", "ipadgotra", "kvogt1", "anton", "lhanopol", "pierre_adrien", "NasserNL", "alexbuijs", "megan_odowd", "Astahn", "zbadgley", "gweaver", "jejacks0n", "kwiebers", "cam_swords", "julianthome", "tnix", "brandon_lyon", "sethgitlab", "dstull", "stkerr", "smanzuik", "mmalcolm", "tmedlin", "adamcohen", "ggillies", "ovider", "jbraughler", "susanhill", "mattkasa", "mrincon", "marc_disabatino", "nelbacha", "cat", "himkp", "kgoossens", "ali-gitlab", "sean_carroll", "sri19", "ssarff", "SChallis", "calbaker", "brian.collins", "esybrant", "V_Rusin", "beckalippert", "timhey", "iamricecake", "pete_goldberg", "jayswain", "sgrahamgl", "cmoberly", "chris_baus", "s_awezec", "kathleentam", "thomasrandolph", "johnhope", "mvrachni", "DebbieHarris", "a_akgun", "mbobin", "sarcila", "kcomoli", "gitter-badger", "ohoral", "ck3g", "rhardarson", "seanarnold", "DPetronio", "rhueston", "mgilgalindo", "cchiappe", "jmuehlbauer", "ljlane", "tparuchuri", "gt1230", "brianglanz", "kevenhughes", "jakeburden", "monica_galletto", "ronkoster", "sstern", "PanosR", "marc_disabatino-admin", "woosang.lee", "niskhakova", "hdevlin", "jackie_fraser", "Weatherford", "mkreuch", "aglidden", "douglasrobbin", "c_weber", "pburdette", "kokeefe", "a.hansen", "eurie", "sawalsh", "nicholasklick", "ademblin", "jblevins608", "djensen", "jheimbuck_gl", "Hortel", "emilyplotkin", "dhershkovitch", "amcmahon88", "mkarampalas", "mbruemmer", "cam.x", "msubramanian", "pcalder", "nsharma2", "nicolasdular", "simon_poon", "nprecilla", "nagyv-gitlab", "arturoherrero", "cngo", "hollyreynolds", "calebw", "nhxnguyen", "jjstark", "laurenbarker", "jrabbits", "SVBarry", "jj-ramirez", "m_gill", "suhegbu", "crennie", "jfeeney", "james_harrison", "nbsmith", "mjwood", "davis_townsend", "tuckcodes", "nmezzopera", "c.dios", "julie.samson", "dmoraBerlin", "jrreid", "npost", "kreykrey", "cbazan1", "abhavadeep", "skarpeles", "caalberts", "bastian_vanderstel", "dzalbo", "mhennessy", "tideker", "david", "bprescott_", "allison.browne", "derekatwood", "Jodij13", "mskumar", "nhouston", "mchadliev", "maria_szucs", "jngrant", "waissaoui", "sladgrove", "doniquesmit", "c.yoshida", "rtakken", "ifrenkel", "asjones", "admiralboom", "DarrenEastman", "jeffcrow", "cdybenko", "Greib", "whaber", "jpetr", "matt_wilson", "fjdiaz", "alexives", "mnevolo", "d0c-s4vage", "cmcclure1", "justin_ho", "nadia_sotnikova", "laurie_howitt", "cfoster3", "daleb04", "jaltheide", "lshand", "rgovender", "Vbrew", "gee_lee", "a_luna", "mkaeppler", "aalfarania", "10io", "lienvdsteen", "pmejia", "FionaOKeeffe", "dfoley092", "mattgonzales", "zcuddy", "sheininger", "JMLeslie", "lmwilliams", "Arashidi", "LoredanaIluca", "nudalova", "joernchen", "mscarnato", "mvilain", "yhsueh", "kdusek", "tabetha.baumechandler", "drivey1", "bgadberry", "jlparker", "rreich", "rmsrohan", "tmorwani", "k_cook", "simon_mansfield", "jdbeaumont", "mfischer37", "DarwinJS", "davegilbert", "libbyschulze", "Ikosenko", "nmilojevic1", "rotanak", "crystalpoole", "oheigre", "cwiesner", "ksundberg", "cherryhan", "jeffersonmartin", "shoyle", "mikolaj_wawrzyniak", "HFrancisco", "wspillane", "gyoachum", "ejbrown71", "joelbeck", "jschafer", "hswimelar", "thaoyeager", "DavidMack", "danielmarquard", "dknopf1", "bpiscitelli", "rragozzine", "supadhyaya", "m_frankiewicz", "aevstifeev", "ncregan", "asalii", "iamphill", "dphelan", "csapolu", "mdewji", "msedlakjakubowski", "Andysoiron", "JDSmits", "fguibert", "mstamper", "amueller327", "m_lussier", "slee24", "tomquirk", "iyoo", "cweaver1", "joshswann", "c_hupy", "jreporter", "michael_lutz", "tshurtleff", "lkerr", "emathis", "bchalfantkero", "brandonbrooks", "dftian", "corina_patachia", "jiaan", "sunjungp", "shreyasagarwal", "m_gore", "RobbieB", "mkulakowski", "jordi_mon", "vicbell", "8bitlife", "ggeorgiev_", "caesarh", "edmondchan", "changzhengliu", "alexpooley", "minsuhan", "jdasmarinas", "ndesilva", "jeromezng", "zmartins", "furkanayhan", "tsaunders", "ngaskill", "ebanks", "kgammon", "KSetschin", "dskim_gitlab", "nutaurus", "djadmin", "derekferguson", "cwoolley-gitlab", "r_kahn", "mjang1", "cmestel", "mwhodges", "dcouture", "magalibressan", "shayfleming", "tav_scott", "kkwentus1", "jhyson", "LindsayOlson", "sfirdaus", "carlos.dominguez", "rhancock", "treagitlab", "lvanc", "pluthra", "plaurinavicius", "Helenadixon", "markkwilliams", "robclark14", "jdrpereira", "chrisanderton", "vdesousa", "dgruzd", "cmayer1", "astoicescu", "ksteffenson", "rkuba", "allison.walker", "tdominique", "malkobaisy", "john_long", "nickbrandt", "pkaldis", "dsearles", "mnearents", "rschulman", "egraves", "PKnorr", "djolsen", "SChia", "jrandazzo", "aseguillon", "hmason", "eirinipan", "MeganWieling", "hilaqu", "glucchesi", "bethpeterson", "JenMcCarthy", "sttruong", "fseifoddini", "hpoojary", "aqualls", "adriansmolski", "cmccarty", "lilphil", "poffey21", "gitlab-dependency-bot", "robotmay_gitlab", "igorwwwwwwwwwwwwwwwwwwww", "ashcampbell", "jeanpeguero", "rickwalker", "rhassanein", "kclaflin", "samcguire", "rachel_hill", "bmiller1", "s_shaik", "vincywilson", "cbednarz", "csouthard", "amandakhughes", "jo_shih", "garyh", "zrice", "sgoldstein", "kbychu", "chilling32", "bryanwise", "sselhorn", "trighetti", "rob.hunt", "ikryzeviciene", "r.leite", "cstahl", "md_silva", "cnicholson", "jsandlin", "saraedavila", "dsamson", "m_walker", "ericschurter", "broncato", "kdemircan", "akiripolsky", "ken_aguilar", "b.singh", "samer.akkoub", "jaime", "svistas", "Quintasan", "jrosendahl", "plaurinavicius-admin", "malkobaisy-admin", "parker_ennis", "kmarquart", "kahmad", "JohnathanHunt", "emilyring", "jordan_ng", "mdunninger", "nabeel.bilgrami", "ricardoamarilla", "vij", "morefice", "jmichniewicz", "hdelalic", "Kgodoy", "mikelong", "kdsouza2", "lstinson", "jmandell", "pwieczorek", "msellers", "pbair", "lcoleman", "rhartough", "joshdowney", "jayo", "kassio", "tancnle", "psureshbabu", "monicaj", "f_caplette", "shemgyll", "wirepair", "shoneycutt", "tmccaslin", "pedropombeiro", "craigmsmith", "ecepulis", "jkenneysmith", "hchristey", "sam.white", "xli1", "nuritzi", "vyaklushin", "alinamihaila", "ralfaro", "atroschinetz", "jfenter", "RDemmer", "manuel.kraft", "nmccorrison", "rhendrick", "goran_bijelic", "ssarka", "mjamison", "brentnewton", "Jon_burghart", "msendal", "justinfarris", "peterhegman", "a_mcdonald", "aseow", "agarciatesares", "kchappell", "abell2", "lmai1", "pgerhold", "fkurniadi", "m_lussier-admin", "hwangui", "oregand", "ashjameson", "idawson", "gitlab_analytics", "sbrower", "amy.waller", "pks-t", "willmeek", "swainaina", "ash2k", "clevelandbledsoe", "andrew_murray", "dnsmichi", "aidanpender", "aturinske", "rickywiens", "jessicareeder", "brufener", "samihiltunen", "svedova", "mparuszewski", "mrleutz", "shaunmccann", "marleyr", "sloyd", "csouleles", "msoares1", "alex_venter", "nicolewilliams", "paul_armstrong", "nwestbury", "dFarnsworth04", "dwainaina", "rchan-gitlab", "annie_huang", "rspainhower", "mclyne", "spatching", "KJaeger", "vkelkar", "scottmckee", "vshumilo", "albertoramos", "cheryl.li", "terrichu", "IanMoore", "susantacker", "JohnMcGuire", "katrinleinweber", "angulol", "rparker2", "mushakov", "streas", "aasimkhan", "johncmarvin", "minac", "Josh_Zimmerman", "mle", "dreedy", "juliebyrne", "l_vazquez", "mbadeau", "mray2020", "cnorris", "warias", "Julia.Lake", "hduncan", "lschoenfeld", "gggonzalez", "kaunghtet", "philipcunningham", "thiagocsf", "mnielsen", "svij", "Courtney_cote", "adawar", "kevinfranklin", "mmederos", "kengrabowski", "jhogue", "ngeorge1", "ysun3", "RFrazier", "pbyrne19", "lasayers", "mwoolf", "s_mccauley", "chloeliu", "awestbrook", "ichiang", "csaavedra1", "msalik", "ajwalker", "JamesRLopes", "vijirao", "r_williams", "bryan-may", "arooke", "nlomas", "ccheong2020", "ypats", "iabbasi", "mscott3", "marc_shaw", "gjudy", "bvanderkolk", "jfullam", "Andjay", "dpduncan", "cs.wang", "laurence.bierner", "cbeer", "ashleypotter", "jlear", "kshirazi", "euko", "mlockhart", "lisapuzar", "tguibert", "mark.art", "sebawebber-admin", "bregan1", "sbouly", "mross1", "KiraAubrey", "bcarranza", "mlunoe", "hsmith-watson", "v_mishra", "mikeeddington", "toshitakaito", "Daryayar", "jayjoshua", "amyphillips", "leosydney", "mpreuss22", "aregnery", "asmolinski2", "bcanton", "jscadden", "ageary", "awthomas", "duncan_harris", "mfitzgerald2", "ageaneas", "cankney", "ccnelson", "sophia_simunec", "rchachra", "cfarris", "alcurtis", "nfiguera", "joebrady", "lkorbasiewicz", "reshmikrishna", "sstalie", "StephenDuggan", "agulina", "dhaines1", "matthewcoughlan", "katieramos", "leetickett", "mmora", "mjozenazemian", "jgianoutsos", "djparker", "godfat-gitlab", "bfoster", "Brock_Sumner", "emorita", "dnolan1", "jonlimr", "kayokocooper", "jonathanvenezia", "aagounder", "m_barnes", "vitorinocoelho", "thornsby", "mmaneval20", "ggraves1", "mthatcher", "calebcooper", "abraun2", "emchang", "adiwolff", "jtogher", "amichell", "avar", "awarden2002", "vikijuhasz", "lpollitt", "klawrence"]
[ gprd ] production> usernames.count
[ gprd ] production> users = User.where(username: usernames).all
[ gprd ] production> users.count
[ gprd ] production> users.each { |u| Feature.enable(:dynamic_image_resizing_owner, u) }
-
Check immediate WH health status
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 240
-
Post in #production (Slack): /chatops run feature set dynamic_image_resizing_requester 5 -
Observe the WH performance: CPU, Memory, latency. Verify that there are no visible changes. -
Check related Prometheus metrics -
Check related logs in Kibana -
Increase the dynamic_image_resizing_requesterpercentage up to 100 (could be done in multiple steps). -
Check WH health, Prometheus, Kibana
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 5
-
Post in #production (Slack): /chatops run feature delete dynamic_image_resizing_requester -
Verify FF is deleted (copy the Slack link https://gitlab.slack.com/archives/C101F3796/p1600183424431100) -
Post in #production (Slack): /chatops run feature delete dynamic_image_resizing_owner -
Verify FF is deleted (copy the Slack link https://gitlab.slack.com/archives/C101F3796/p1600183437431300)
Monitoring
Key metrics to observe
- Metric: GL Health
- Location: https://dashboards.gitlab.net/d/RZmbBr7mk/gitlab-triage?orgId=1&refresh=30s
- What changes to this metric should prompt a rollback: Error rate growth, significant response time growth
- Metric: gitlab_workhorse_image_resize_concurrency_limit_exceeds_total
- Location: https://prometheus-app.gprd.gitlab.net/graph
- What changes to this metric should prompt a rollback: Nothing
- Metric: gitlab_workhorse_image_resize_processes
- Location: https://prometheus-app.gprd.gitlab.net/graph
- What changes to this metric should prompt a rollback: No
- Metric: gitlab_workhorse_image_resize_completed_total
- Location: https://prometheus-app.gprd.gitlab.net/graph
- What changes to this metric should prompt a rollback: value remains zero after
dynamic_image_resizing_owneris enabled for all users
- Metric:
ImageResizer: Successlog record- Location: https://log.gprd.gitlab.net/goto/53283b48d53ba14934a46410d9412bb7
- What changes to this metric should prompt a rollback: zero records during the first 30 mins of the experiment
- Metric:
ImageResizer:...log records (various errors)- Location: https://log.gprd.gitlab.net/goto/53283b48d53ba14934a46410d9412bb7
- What changes to this metric should prompt a rollback: > 1000 error records of any type
- Metric: Workhorse/Full web component Health; WH CPU & Memory usage
- Location: https://dashboards.gitlab.net/d/web-main/web-overview?orgId=1 and https://dashboards.gitlab.net/d/general-service/general-service-platform-metrics?orgId=1
- What changes to this metric should prompt a rollback: Error rate growth, significant response time growth
- Metric: WH request CPU usage
- Location: https://log.gprd.gitlab.net/goto/53283b48d53ba14934a46410d9412bb7
- What changes to this metric should prompt a rollback: Nothing
- Example of the report: https://gitlab.com/gitlab-org/gitlab/-/issues/241533#note_409961524
Summary of infrastruture changes
- [-] Does this change introduce new compute instances? - No
- [-] Does this change re-size any existing compute instances? - No
- [-] Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc? - No
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled). -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and resultes noted in a comment on this issue: https://gitlab.com/gitlab-org/gitlab/-/issues/241533#note_407403888 -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncalland this issue.) -
There are currently no active incidents.
Edited by Aleksei Lipniagov