All snippets have line endings converted to CR/LF

To clarify, the very reason I noticed this in the first place is that I have been transitioning from GitHub to GitLab, and as such the above is my workflow to temporarily use gists/snippets i've saved. Bash does not parse this correctly due to the CR/LF line endings and thus it breaks the script.

Has it always been this way or is this a regression from one of the many recent updates?

Added ~13817 label

Anyone?

mentioned in issue #21556 (closed)

Added ~874211 label

I've noticed this as well, it's frustrating. I also resorted to using dos2unix to convert the downloaded snippets and execute. GitHub does not convert "gist" line endings.

Added ~313020 label

@jschatz1 Here is the ticket that made me add the customer+ https://gitlab.zendesk.com/agent/tickets/46029

This has proven to be a pain point which caused me to avoid snippets altogether. I was attempting to host a shell script whose url I was passing to a downstream utility. Since I have no control over the actual download and execution commands, I cannot convert the line endings on the fly. Therefore, I am forced to go with a different solution entirely. Please address this issue!

Also, the zendesk issue linked above comes up 404 for me.

added ~1071781 label

@ajk8 Thanks for your feedback. The Zendesk issue is internal only. All relevent information is copied to the GitLab issue.

^^^ not me, but @ajk8

same issue here, I use like bash <(curl -sL https://gitlab.com/snippets/33722/raw) to get system info or other kind of quick installs or scripts

Although I'm a big supporter of gitlab, I'll have to put the snippet on github till this is resolved.

I am evaluating GitLab for an enterprise replacement of GitHub, and am happy to pay for the enterprise edition, but this is sort of a big issue for us. We have the same use case as @dgoo2308. I would also love to see versioning for snippets, but I understand that may be a way off.

Meanwhile, I'll take a look at the source to see if there is an easy way to at least add an argument to the /raw action like /raw?line-endings=LF or similar.

I'm not a Ruby developer, but here's a workaround for the time being:

kamermans/gitlab-ce@911b99bf

Edit app/controllers/snippets_controller.rb and replace the definition of the def action with this:

  def raw
    content = @snippet.content
    if params[:lineendings] == "lf"
      content = content.gsub(/\r\n/m, "\n")
    end
    send_data(
      content,
      type: 'text/plain; charset=utf-8',
      disposition: 'inline',
      filename: @snippet.sanitized_file_name
    )
  end

Then you will be able to add ?lineendings=lf to your /raw URL and it will return LF line endings instead of CRLF. I think the root problem is that the HTML spec for textareas causes the line endings to be stored as CRLF to begin with, so I'm not sure what a better alternative is, besides, perhaps, making the default line endings LF and allowing the user to change to between LF <=> CRLF in the GUI when editing a snippet.

I can send a pull request if you guys want, but it seems like this should be handled by making LF the default.

mentioned in issue #26515 (closed)

If the root of the problem here is that form-data is sent with CRLF line endings, as suggested by @Necrathex in issue #26515 (closed), then I would suggest encoding the content before sending it to the server (in base64, for example), so the original data is preserved.

Still, it might be nice to have a toggle switch on the editor to let you choose the line endings for your snippet. This would accommodate the case where a Windows user wants to copy and paste a UNIX script into a new snippet with LF line endings.

I agree with @kamermans that having the original data preserved would be optimal, but I'm not sure that is possible since the html5 spec states that the textarea contents will contain only LF characters (if I understand correctly) so that would strip CR chars for Windows file snippets instead?

From the spec:

the user agent should allow the user to edit, insert, and remove text, and to insert and remove line breaks in the form of "LF" (U+000A) characters

In that case a UI switch would be great.

Probably best to just emulate whatever github is doing here.... They got it right, in this case.

/cc @victorwu @smcgivern

Good catch @Necrathex, I didn't realize that was in the spec.

Personally I would just force every thing to LF anyway, as I've never seen a file break anything with just LF line endings in 20 years of being a sysadmin and developer, except on really old Macs. Windows machines are totally fine with LF unless you're using Notepad. Even converting an entire Microsoft Visual C# project to LF line endings does not break it.

Thanks everyone for the input!

@Necrathex: that part of the spec describes editing. For submission, this applies: https://html.spec.whatwg.org/multipage/forms.html#textarea-wrapping-transformation

The textarea wrapping transformation is the following algorithm, as applied to a string:

Replace every occurrence of a U+000D CARRIAGE RETURN (CR) character not followed by a U+000A LINE FEED (LF) character, and every occurrence of a U+000A LINE FEED (LF) character not preceded by a U+000D CARRIAGE RETURN (CR) character, by a two-character string consisting of a U+000D CARRIAGE RETURN U+000A LINE FEED (CRLF) character pair.

[...]

@xenithorb: emulating GitHub in this case would involve making our snippets backed by a repository, which is a great idea but is a lot of work for this change. Line endings inserted through the web editor in a repository on GitLab work fine, which I think - I haven't double-checked - is because of git's normalisation.

Am I right in saying that the minimum acceptable change for everyone here is to do something like @kamermans suggested in https://gitlab.com/gitlab-org/gitlab-ce/issues/21240#note_21137076 - that is, add an extra param?

I'm not sure how we'd expose that in the UI, or otherwise document it, but that seems reasonable to me.

Note that another workaround may be to use the API to create or update snippets, as that won't be constrained by the browser's form-encoding requirements. I realise that's not a great workaround, just mentioning it

@smcgivern I've actually went around the UI entirely and jammed my snippet into PostgreSQL without CRLF, but somehow it was still served with CRLF. I suspect there is some sort of filter upstream in the output code.

Here are my preferred solutions at this point, in order from most-preferred to least-preferred (but still usable):

Save and serve all snippets with LF line endings by default, but have a switch in the GUI to select the preferred line endings for the snippet.
Save and serve all snippets with LF line endings. Add a parameter ?lineendings=<lf|crlf> so it can be forced.
Save and serve all snippets with LF line endings and do not support CRLF.

I any case, I think the default should be changed from CRLF to LF, especially considering there is no way to know what the original line endings were.

The absolute minimum that needs to happen here is that it should default to LF line endings only.

If you could get it to honor the original input, that would be the most optimal I would think, but something was mentioned about a specification hindering that as I understand it.

My personal take is that I think it would be such an infrequently used feature, that @kamermans number 2) point is probably the best option and quickest and easiest to implement, while still allowing an override if needed.

That being said if the original data can be maintained then the options are moot and that is what should happen.

Could it be done in such a way that the browser encapsulates the data before it's transmitted as to evade the specification? My apologies if i'm not articulating that properly web development is not my area. Thanks

I've created a little test page that demonstrates that my suggestion of base64-encoding the data before submitting it, then decoding it in the response does in fact preserve the original line-endings:

Live Demo: http://srkdev.com/gitlab/test.php

Source Code: https://gist.github.com/kamermans/73e4889e486b5222dbba464c156d8d7b

In the demo, the textarea is data, then there is a hidden field under it called encoded_data. When you submit the form, I run btoa() on data and save it to encoded_data, then submit the form. On the server side, I receive the form (in PHP) and base64_decode() encoded_data and save it in a new field called decoded_data, then output the contents in JSON so we can see the line endings in the output.

(sorry for using GitHub, but I wanted to have versioning and LF line endings :P)

@kamermans are you sure about the test you did? Maybe I'm misunderstanding, but it seems to work for me:

$ curl -H "Private-Token: $api_token" -H 'Content-Type: application/json' -X POST https://gitlab.com/api/v3/snippets -d '{"title":"Test snippet","file_name":"test.sh","visibility_level":20,"content":"echo Hello\necho World"}
{"id":35319,"title":"Test snippet","file_name":"test.sh","author":{"name":"Sean McGivern","username":"smcgivern","id":443319,"state":"active","avatar_url":"https://secure.gravatar.com/avatar/05ae1e8818aa4a3333ab7f72acfb20c0?s=80&d=identicon","web_url":"https://gitlab.com/smcgivern"},"updated_at":"2017-01-13T11:40:48.156Z","created_at":"2017-01-13T11:40:48.156Z","web_url":"https://gitlab.com/snippets/35319","raw_url":"https://gitlab.com/snippets/35319/raw"}
$ curl -s https://gitlab.com/snippets/35319/raw | sh
Hello
World

I think @xenithorb's suggestion of just converting to LF all the time, and adding a param to keep CRLF, is the simplest. Otherwise the behaviour will depend on which input form you used (the old version or the new version), which is quite unpredictable. We can add a note to https://docs.gitlab.com/ee/workflow/project_features.html#snippets explaining this.

added Deliverable label

changed milestone to %8.17

@smcgivern I'm not 100% sure about my test, I was using socat to expose the pgsql UNIX domain socket to a TCP port so I could sneak into the DB and poke around. I could easily have made a mistake while running that test.

I support the fix you've suggested.

@smcgivern: I've confirmed that the API does preserve LF and serve it back as LF.

Thanks @kamermans, I've updated the description to confirm the solution that we're going to go with

assigned to @jarka

mentioned in merge request !8999 (merged)

Unfortunately this didn't quite make the feature freeze for 8.17, but it will be in 9.0!

changed milestone to %9.0

closed via merge request !8999 (merged)

added priority3 severity3 and removed ~1672339 labels

Issue is still exists (CE 11.5) when using raw get with private token param.

curl -sS 'https://example.ert.com/api/v4/snippets/20/raw?private_token=xxx'

mentioned in issue #56901 (moved)

@roman.vasilev is right, this issue still exists on Gitlab 11.9.6-ee when you download the content of a snippet with the API (v4).

I checked the documentation (https://docs.gitlab.com/ee/api/snippets.html) but I haven't found any way to fix that.

Gitlab 11.11.2-ce still have this issue when snippet is created with Gitlab UI/textarea.

added devopsplan label

marked this issue as related to #44032 (moved)

mentioned in issue gitlab#26379 (closed)

changed the description

removed time estimate

added Category:Source Code Management snippets labels and removed 1 deleted label

All snippets have line endings converted to CR/LF

Problem

Solution

Original description

Designs

Child items ...

Activity

All snippets have line endings converted to CR/LF

Problem

Solution

Original description

Relates to

Activity