AWS S3 API doesn't work with FIFO
// From: Thomas
I've tried to configure the FIFO version 0.9.1 with AWS S3 to use it for images and backups. But it doesn't work and return an error in the {{sniffle}} log:
2017-05-04 17:42:25 =ERROR REPORT====
** State machine <0.25495.0> terminating
** Last event in was timeout
** When State == get_manifest
** Data == {state,undefined,<<"https://imgapi.smartos.skylime.net/images/156cac3d-2d48-489e-b12c-5ff493bffe2d">>,undefined,undefined,undefined,undefined,undefined,undefined,<0.3029.0>,#Ref<0.0.1.42207>,<<>>,0,[],undefined,false}
** Reason for termination =
** {{badmatch,{error,{error,{http_error,400,"Bad Request",<<"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message><RequestId>80BEDA13F16391B1</RequestId><HostId>INMtLKaD1O0DbIcCnWUR40vMCjKidFhQM6n2AmFqlE0c+G9YEpMYlApkOJ7x9iTI/IkWelm4bz0=</HostId></Error>">>}}}},[{sniffle_dataset_download_fsm,get_manifest,2,[{file,"/root/workspace/Fifo_15.4.1_Rel/Sniffle-15.4.1-Rel/_build/prod/lib/sniffle_dataset_download/src/sniffle_dataset_download_fsm.erl"},{line,128}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,518}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
2017-05-04 17:42:25 =CRASH REPORT====
crasher:
initial call: sniffle_dataset_download_fsm:init/1
pid: <0.25495.0>
registered_name: []
exception exit: {{{badmatch,{error,{error,{http_error,400,"Bad Request",<<"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message><RequestId>80BEDA13F16391B1</RequestId><HostId>INMtLKaD1O0DbIcCnWUR40vMCjKidFhQM6n2AmFqlE0c+G9YEpMYlApkOJ7x9iTI/IkWelm4bz0=</HostId></Error>">>}}}},[{sniffle_dataset_download_fsm,get_manifest,2,[{file,"/root/workspace/Fifo_15.4.1_Rel/Sniffle-15.4.1-Rel/_build/prod/lib/sniffle_dataset_download/src/sniffle_dataset_download_fsm.erl"},{line,128}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,518}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},[{gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,626}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
ancestors: [sniffle_dataset_download_sup,<0.247.0>]
messages: []
links: [<0.248.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 1598
stack_size: 27
reductions: 14185
neighbours:
2017-05-04 17:42:25 =SUPERVISOR REPORT====
Supervisor: {local,sniffle_dataset_download_sup}
Context: child_terminated
Reason: {{badmatch,{error,{error,{http_error,400,"Bad Request",<<"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message><RequestId>80BEDA13F16391B1</RequestId><HostId>INMtLKaD1O0DbIcCnWUR40vMCjKidFhQM6n2AmFqlE0c+G9YEpMYlApkOJ7x9iTI/IkWelm4bz0=</HostId></Error>">>}}}},[{sniffle_dataset_download_fsm,get_manifest,2,[{file,"/root/workspace/Fifo_15.4.1_Rel/Sniffle-15.4.1-Rel/_build/prod/lib/sniffle_dataset_download/src/sniffle_dataset_download_fsm.erl"},{line,128}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,518}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
Offender: [{pid,<0.25495.0>},{id,sniffle_dataset_download_fsm},{mfargs,{sniffle_dataset_download_fsm,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]
The following configuration is used:
storage.s3.host s3-eu-central-1.amazonaws.com
storage.s3.port 443
storage.s3.access_key ****
storage.s3.secret_key ****
storage.s3.general_bucket tm-fifo
storage.s3.image_bucket tm-fifo-images
storage.s3.snapshot_bucket tm-fifo-backups
LeoFS is the suppored s3 storage used by fifo, while AWS is nice-to-have it's not part of the core functionality. Will investigate when I got some free time and nerves to mess with AWS
Maybe the current API is already supported with the newest https://github.com/erlcloud/erlcloud version?
Yea I'm trying that right now, or at least building a dev chunter wit it for you to try
If you want to see if it works with dev? If it does (and also works with Leo) it'd be an easy fox
It would be nice if you could provide me with the correct URL I should use to receive the dev packages. I'm not sure if http://release.project-fifo.net/pkg/15.4.1/dev is correct.
https://docs.project-fifo.net/docs/chunter#section-installation details how to install a dev version.
Ohh thanks, I thought it requires a new sniffle version not a chunter version
Could be, but best to try changing as little things at once as possible
I've updated to the dev version but I sill receive the error, so I assume I'm sorry a version bump of erlcloud doesn't fix it
2017-05-07 12:00:56.336 [error] <0.2275.0> CRASH REPORT Process <0.2275.0> with 0 neighbours exited with reason: {error,{http_error,400,"Bad Request",<<"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message><RequestId>07F352323EFEA4A0</RequestId><HostId>0OLx1UF8XxkdBGgYUtVVcL+BzGuwECWxOa8EZEHKqoQUBgMu9yHIQZDpxjuEpqUi/bm37uhTvJA=</HostId></Error>">>}} in gen_server:init_it/6 line 344
2017-05-07 12:00:56.337 [error] <0.2221.0> gen_fsm <0.2221.0> in state get_manifest terminated with reason: no match of right hand value {error,{error,{http_error,400,"Bad Request",<<"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message><RequestId>07F352323EFEA4A0</RequestId><HostId>0OLx1UF8XxkdBGgYUtVVcL+BzGuwECWxOa8EZEHKqoQUBgMu9yHIQZDpxjuEpqUi/bm37uhTvJA=</HostId></Error>">>}}} in sniffle_dataset_download_fsm:get_manifest/2 line 128
2017-05-07 12:00:56.341 [error] <0.2221.0> CRASH REPORT Process <0.2221.0> with 0 neighbours exited with reason: no match of right hand value {error,{error,{http_error,400,"Bad Request",<<"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message><RequestId>07F352323EFEA4A0</RequestId><HostId>0OLx1UF8XxkdBGgYUtVVcL+BzGuwECWxOa8EZEHKqoQUBgMu9yHIQZDpxjuEpqUi/bm37uhTvJA=</HostId></Error>">>}}} in sniffle_dataset_download_fsm:get_manifest/2 line 128 in gen_fsm:terminate/7 line 626
2017-05-07 12:00:56.341 [error] <0.248.0> Supervisor sniffle_dataset_download_sup had child sniffle_dataset_download_fsm started with {sniffle_dataset_download_fsm,start_link,undefined} at <0.2221.0> exit with reason no match of right hand value {error,{error,{http_error,400,"Bad Request",<<"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message><RequestId>07F352323EFEA4A0</RequestId><HostId>0OLx1UF8XxkdBGgYUtVVcL+BzGuwECWxOa8EZEHKqoQUBgMu9yHIQZDpxjuEpqUi/bm37uhTvJA=</HostId></Error>">>}}} in sniffle_dataset_download_fsm:get_manifest/2 line 128 in context child_terminated
Is it somehow easy possible to add some debugging option to see the exact request?