Backup restore SKIP option (omnibus)
Description
Hi there!
I'm trying to restore backup and sometimes it fails due to my mistakes (I choose wrong filesystem for, or give it too small fs to restore to). And it's ok, but it fails on some step like restoring artifacts
(when db, repositories and other steps are passed normaly). And so, when I see it fails, for example because of no space left on device
- I want resize my filesystem and run backup from the step it failed (artifacts
) without restoring other parts that were already restored succsessfuly. I saw in documentation that there is SKIP
option during create
operation, but it seems like it doesn't work with restore
, when I try gitlab-rake gitlab:backup:restore SKIP=db,repositories,lfs BACKUP=$BACKUP_NAME
it doesn't take any effect, it starts to unpack whole backup (which is painfully with big installations and slow disk subsystem) and ask me if I want to restore my database again.
Proposal
SKIP
option should also work with gitlab-rake gitlab:backup:restore
. Also, as I remember it's possible to untar only those files from archive that you need, it would be nice if restore
will unpack only those files and directories that is not SKIP
'ed
Documentation blurb
Overview
It would be useful when you have corrupted your gitlab data and need to restore it ASAP but you meet some problems during restoration like no space left on device
, or read-only fs
, or something like "unreachable remote block device" (if you use remote storage) at one of the last steps and start restoring again after fixing some underlying infrastructure problems.
Feature checklist
Make sure these are completed before closing the issue, with a link to the relevant commit.
- Feature assurance
- Documentation
- Added to features.yml
Designs
- Show closed items
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- Developer
@homelessnessbo1 the
gitlab:backup:restore
task is a meta task which consists of other tasks. You can see it here: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/tasks/gitlab/backup.rake#L56-64You can run any task from that list manual like
gitlab-rake gitlab:backup:TASK_NAME_HERE
.Does it make sense?
- Author
I'll check it today, thx
- Author
@blackst0ne thank you for your response, but when I try to make it the way you suggested, I get this:
root@gitlab-ce:/# gitlab-rake gitlab:backup:builds:restore --trace BACKUP=1503524805_2017_08_24_9.4.1 ** Invoke gitlab:backup:builds:restore (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute gitlab:backup:builds:restore rake aborted! NoMethodError: private method `puts' called for nil:NilClass Did you mean? putc /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:120:in `block (4 levels) in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:240:in `block in execute' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `execute' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:179:in `block in invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:172:in `invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:165:in `invoke' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:150:in `invoke_task' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `block (2 levels) in top_level' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `block in top_level' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:115:in `run_with_threads' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:100:in `top_level' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:78:in `block in run' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:176:in `standard_exception_handling' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:75:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/bin/rake:33:in `<top (required)>' /opt/gitlab/embedded/bin/rake:22:in `load' /opt/gitlab/embedded/bin/rake:22:in `<top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `load' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `kernel_load' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in `exec' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in `dispatch' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in `start' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:34:in `block in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in `with_friendly_errors' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:26:in `<top (required)>' /opt/gitlab/embedded/bin/bundle:22:in `load' /opt/gitlab/embedded/bin/bundle:22:in `<main>' Tasks: TOP => gitlab:backup:builds:restore
I'm running gitlab with official image
gitlab/gitlab-ce:9.4.1-ce.0
. Any thoughts? - Author
And as I can see there is no huge difference between 9.4.1 and current master branches in that file https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/tasks/gitlab/backup.rake
- Developer
@homelessnessbo1 yeah, sorry, thought it could work.
By the way, did you take a look at this docs? https://docs.gitlab.com/ce/raketasks/backup_restore.html#excluding-specific-directories-from-the-backup
It seems this is what you want.
- Author
@blackst0ne Actualy, according to the code you provided me link to in the first commnt - I think it supposed to work and it looks like there is some bug with
gitlab-rake gitlab:backup:builds:restore
(and may be with other sub-commands). And yes, I've read all this docs from your last comment and as I mentioned in initial issue description - I've already tried to useSKIP=...
and it doesn't take any effect withrestore
subcommand. Any news about this topic ? I have also the same kind of issue. I would like to restore only what i backed up. For example, i daily backup with skipping artifacts & registry to if i restore with this backup i don't want to remove my current artifacts & registry data.
- Douwe Maan added gitlab-ce~24926493 label
added gitlab-ce~24926493 label
- Contributor
ZD https://gitlab.zendesk.com/agent/tickets/90406 (internal)
https://gitlab.com/gitlab-org/gitlab-ce/issues/37312#note_39211931 is essentially a duplicate of https://gitlab.com/gitlab-org/gitlab-ce/issues/40617, which will be fixed by https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19122.
- Author
@stanhu yo, thanks for letting know! This problem is still actual and fixing it would make our live a bit easier.
@stanhu - I don't think this is related to gitlab-ce#40617 as that is a bug. This appears to be a feature request to allow the skipping of directory processing even if they were not skipped during backup. Currently backups contain a file backup_information.yml which notes what was skipped during backup. I believe the request here is to add to that list (or create it if not in the yml) during a restore, no matter what SKIP options were specified during the backup.
I'll just put this here in case anyone stumbles across this looking to do something similar. I was having trouble restoring from a backup wherein the 'registry' step was failing [still unsure why]. So, I wanted to skip it and the
SKIP=registry
flag did not work? This did (repeat or add other-e
flags for desired steps), prior to executinggitlab-rake gitlab:backup:restore
.# Comments out the 'registry' restoration step sed -e '/registry:restore/ s/^#*/#/' \ -i /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake
Of course, uncomment when you're done restoration.
Edited by Zach van Rijn- 🤖 GitLab Bot 🤖 added gitlab-ce
3857529a gitlab-ce7638922 labelsadded gitlab-ce
3857529a gitlab-ce7638922 labels - 🤖 GitLab Bot 🤖 added Create [DEPRECATED] Manage [DEPRECATED] labels and removed 1 deleted label
added Create [DEPRECATED] Manage [DEPRECATED] labels and removed 1 deleted label
@zv-io I have the same problem. In my case, I didn't have any images saved in registry. So I think when there are no images, the file registry.tar.gz is not generated, and it causes this fail.
Thanks for you solution!
- 🤖 GitLab Bot 🤖 added [deprecated] Accepting merge requests label
added [deprecated] Accepting merge requests label
- 🤖 GitLab Bot 🤖 removed [deprecated] Accepting merge requests label
removed [deprecated] Accepting merge requests label
- Maintainer
Setting devopsenablement groupgeo based on ~"backup-restore".
- 🤖 GitLab Bot 🤖 added devopssystems groupgeo labels
added devopssystems groupgeo labels
- 🤖 GitLab Bot 🤖 mentioned in issue #33987 (closed)
mentioned in issue #33987 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #34495 (closed)
mentioned in issue #34495 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #35058 (closed)
mentioned in issue #35058 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #35511 (closed)
mentioned in issue #35511 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #36007 (closed)
mentioned in issue #36007 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #36597 (closed)
mentioned in issue #36597 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #37212 (closed)
mentioned in issue #37212 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #37891 (closed)
mentioned in issue #37891 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #39031 (closed)
mentioned in issue #39031 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #103433 (closed)
mentioned in issue #103433 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #121512 (closed)
mentioned in issue #121512 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #193058 (closed)
mentioned in issue #193058 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #195131 (closed)
mentioned in issue #195131 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #196496 (closed)
mentioned in issue #196496 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #197832 (closed)
mentioned in issue #197832 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #199015 (closed)
mentioned in issue #199015 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #201684 (closed)
mentioned in issue #201684 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #204688 (closed)
mentioned in issue #204688 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #206852 (closed)
mentioned in issue #206852 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #207788 (closed)
mentioned in issue #207788 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #208649 (closed)
mentioned in issue #208649 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #209732 (closed)
mentioned in issue #209732 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #211308 (closed)
mentioned in issue #211308 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #212132 (closed)
mentioned in issue #212132 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #213513 (closed)
mentioned in issue #213513 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #214202 (closed)
mentioned in issue #214202 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #214973 (closed)
mentioned in issue #214973 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #215839 (closed)
mentioned in issue #215839 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #216433 (closed)
mentioned in issue #216433 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #217292 (closed)
mentioned in issue #217292 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #218213 (closed)
mentioned in issue #218213 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #218956 (closed)
mentioned in issue #218956 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #219708 (closed)
mentioned in issue #219708 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #220742 (closed)
mentioned in issue #220742 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #222243 (closed)
mentioned in issue #222243 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #223665 (closed)
mentioned in issue #223665 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #225142 (closed)
mentioned in issue #225142 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #226970 (closed)
mentioned in issue #226970 (closed)
- Developer
- Davin Walker mentioned in merge request !36738 (closed)
mentioned in merge request !36738 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #230667 (closed)
mentioned in issue #230667 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #232390 (closed)
mentioned in issue #232390 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #233344 (closed)
mentioned in issue #233344 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #235099 (closed)
mentioned in issue #235099 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #237837 (closed)
mentioned in issue #237837 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #240868 (closed)
mentioned in issue #240868 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #243481 (closed)
mentioned in issue #243481 (closed)
- 🤖 GitLab Bot 🤖 added sectioncore platform label
added sectioncore platform label
- 🤖 GitLab Bot 🤖 mentioned in issue #246473 (closed)
mentioned in issue #246473 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #249075 (closed)
mentioned in issue #249075 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #254176 (closed)
mentioned in issue #254176 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #257801 (closed)
mentioned in issue #257801 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #262007 (closed)
mentioned in issue #262007 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#504 (closed)
mentioned in issue gitlab-org/quality/triage-reports#504 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#562 (closed)
mentioned in issue gitlab-org/quality/triage-reports#562 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#638 (closed)
mentioned in issue gitlab-org/quality/triage-reports#638 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#736 (closed)
mentioned in issue gitlab-org/quality/triage-reports#736 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#824 (closed)
mentioned in issue gitlab-org/quality/triage-reports#824 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#858 (closed)
mentioned in issue gitlab-org/quality/triage-reports#858 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#932 (closed)
mentioned in issue gitlab-org/quality/triage-reports#932 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#1039 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1039 (closed)
- 🤖 GitLab Bot 🤖 added vintage label
added vintage label
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#1090 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1090 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#1168 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1168 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#1267 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1267 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#1346 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1346 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#1415 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1415 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#1501 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1501 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#1590 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1590 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#1683 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1683 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#1788 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1788 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#1884 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1884 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#1968 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1968 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#2033 (closed)
mentioned in issue gitlab-org/quality/triage-reports#2033 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#2134 (closed)
mentioned in issue gitlab-org/quality/triage-reports#2134 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#2238 (closed)
mentioned in issue gitlab-org/quality/triage-reports#2238 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#2332 (closed)
mentioned in issue gitlab-org/quality/triage-reports#2332 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#2398 (closed)
mentioned in issue gitlab-org/quality/triage-reports#2398 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#2497 (closed)
mentioned in issue gitlab-org/quality/triage-reports#2497 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#2563 (closed)
mentioned in issue gitlab-org/quality/triage-reports#2563 (closed)
- Nick Nguyen changed milestone to %Backlog
changed milestone to %Backlog
- 🤖 GitLab Bot 🤖 added [deprecated] Accepting merge requests label
added [deprecated] Accepting merge requests label
- Priyan Sureshbabu mentioned in merge request gitlab-org/charts/gitlab!2305 (closed)
mentioned in merge request gitlab-org/charts/gitlab!2305 (closed)
- Vincent Hindriksen mentioned in issue #19256
mentioned in issue #19256
- Developer
@ibaum @brodock Any thoughts on the effort required to implement this? /cc @sranasinghe @DarwinJS
Collapse replies I don't know about the effort but I do know there is absolutely massive GitLab SRE value in at least two ways:
- Constructing optimized restore with one backup - restore only Git repos first and in a second run the rest.
- Constructing restore scenarios that differentiate from compliance backup requirements. I might be required by compliance to backup artifacts and logs, but have no interest in restoring them in an actual restore event.
- Avoiding late, unsavory discoveries - like my restore is going to take 16 hours because I backup everything and don't need to restore it all. Which could be quite common given that, when you don't know otherwise most of us default to "backup everything".
- Maintainer
The method that needs modifying is here https://gitlab.com/gitlab-org/gitlab/-/blob/3dbb300dcffc8dd57fcf0c00b8805801299ad390/lib/backup/manager.rb#L378
We effectively want to check
ENV['SKIP']
first before we bother checking what is skipped within thebackup_information.yml
. It should be possible to use the same logic for both creating and restoring.This could be a good candidate for a first time contributor.
Edited by James Fargher
- James Fargher mentioned in merge request !83783 (merged)
mentioned in merge request !83783 (merged)
- Maintainer
Added with !83783 (merged)
- James Fargher added workflowstaging label
added workflowstaging label
- James Fargher assigned to @proglottis
assigned to @proglottis
- James Fargher closed
closed
- James Fargher changed milestone to %14.10
changed milestone to %14.10
- James Fargher added workflowproduction label and removed workflowstaging label
added workflowproduction label and removed workflowstaging label
- James Fargher mentioned in issue gitaly#4176 (closed)
mentioned in issue gitaly#4176 (closed)
- James Fargher mentioned in merge request !86081 (merged)
mentioned in merge request !86081 (merged)