Skip to content

Replace mimemagic dependency

As a solution to gitlab-com/gl-infra/production#4054 (closed) we should replace our own dependency on mimemagic.

Options for replacement

Shrine uses the unix file utility by default

I'm actively using this at the moment on my own projects. It does give us a dependency on another system component that we'd have to ensure is installed, probably adding it to our requirements documentation, but it should be installed on all Linux distros by default.

Rails has an active PR to swap to mime-types

I've used this in the past and it worked well, but appears to be less in-depth than mimemagic. Our direct use-cases are limited so this possibly doesn't matter

ruby-filemagic

Shrine also supports this. Wraps libmagic.

Carrierwave only has carrierwave-mimetype-fu

This is defunct

Where we currently use mimemagic

hipchat gem dependency

We might need to fork and modify this gem, or accelerate our removal of it (currently scheduled for 14.0 #27954 (closed)).

Per #325851 (comment 537153041), we will remove the hipchat gem.

PM: @deuley; BE EM: @mnohr; FE EM: @leipert

Direct dependencies

First introduced

$ grep -Rn 'MimeMagic\.'
app/uploaders/content_type_whitelist.rb:46:          MimeMagic.by_magic(file).try(:type) || 'invalid/invalid'
app/controllers/projects/jobs_controller.rb:229:    mime_type = MimeMagic.by_magic(raw_data)
spec/support/shared_contexts/upload_type_check_shared_context.rb:16:    magic_mime_obj = MimeMagic.new(mime_type)

Rails/ActiveStorage

ActiveStorage uses it as a dependency, but we don't use ActiveStorage, so we can stop loading ActiveStorage.

Update: We already exclude this from being loaded, see config/application.rb

It is however still pulled in as a dependency in our Gemfile.lock, probably due to Rails.

Update 1 (2021-03-24)

We have a "working" MR in !57387 (merged), but file is an unreliable dependency across systems, as seen in !57387 (comment 536948103). Ubuntu 18.04, for example, fails to detect a lot of mime types correctly.

@WarheadsSE, @digitalmoksha, and I had a chat about our options. Currently it looks like forking ruby-filemagic or ruby-magic and vendoring a fixed version of file/libmagic into it as the most reliable option, where we can guarantee that the mime types match for everyone using the gem.

We don't believe https://github.com/mime-types/ruby-mime-types is viable as it's based solely off the file extension.

We've cloned the repo over to https://gitlab.com/gitlab-org/ruby-magic and are working to vendor http://www.darwinsys.com/file/ into it so that the fixed version is compiled with the gem.

If anyone has experience with vendoring library dependencies for C-extensions in Ruby, please get involved 😄 Thanks Stan & Yorick

Update 2 (2021-03-25)

We're now using https://github.com/kwilczynski/ruby-magic in master 🎉 Thanks very much to its author Krzysztof Wilczyński for being so responsive and for accepting our changes to have it vendor the file dependencies 😃

The current state:

  • Our own dependencies on mimemagic are gone: !57387 (merged)
  • We're about to merge a shim to override any dependency that still references mimemagic (i.e. Rails): !57443 (merged)
  • These changes are marked for picking so they'll be picked up in our next backport releases
  • We're running into CI issues due to the ruby-magic gem now downloading from public file mirrors. We're swapping to an internal fork at https://gitlab.com/gitlab-org/ruby-magic that uses our own mirror of the archive, and working on a more permanent solution. Fork swap: !57487 (merged)
  • We're waiting for the latest builds to finish to confirm the issue is resolved
  • Builds currently blocked due to git being used in the gemspec, MR to fix this: !57516 (merged). Now fixed.

Apps we're shimming with !57443 (merged):

  1. https://gitlab.com/gitlab-services/version-gitlab-com - requires GPG commit signing to push to the repo
  2. https://gitlab.com/gitlab-org/customers-gitlab-com: https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/2994, git fix MR
  3. https://gitlab.com/gitlab-org/license-gitlab-com: https://gitlab.com/gitlab-org/license-gitlab-com/-/merge_requests/256, git fix MR
  4. https://gitlab.com/gitlab-com/people-group/peopleops-eng/compensation-calculator/ MR, git fix MR
  5. https://gitlab.com/gitlab-com/people-group/peopleops-eng/assessment-tool MR, git fix MR
  6. https://gitlab.com/gl-technical-interviews/backend/template/template-project-clean - MR
  7. https://gitlab.com/gitlab-com/people-group/peopleops-eng/employment-automation/ MR, git fix not needed
  8. https://gitlab.com/gitlab-com/people-group/peopleops-eng/nominatorbot MR, git fix not needed
  9. https://gitlab.com/gitlab-com/www-gitlab-com (dependency through greenhouse_io) MR, git fix not needed
  10. https://gitlab.com/gitlab-org/project-templates/rails - Opened #326177 (closed)

Other todos:

  1. Mirror the file package dependency so we don't nobble their mirrors: ruby-magic!3 (comment 537513966). In progress at https://gitlab.com/gitlab-org/file Done: https://gitlab.com/gitlab-org/file-mirror/-/packages. Automates using the CI when a new release is added.
  2. Swap the mirror in our fork of ruby-magic: ruby-magic!6 (merged)
  3. Currently investigating whether we can just vendor the archive in the gem: https://github.com/kwilczynski/ruby-magic/issues/8#issuecomment-806771710. Legal issue: https://gitlab.com/gitlab-com/legal-and-compliance/-/issues/453
Edited by Thong Kuah