Allow compressed checkpoints in Duo workflows internal API endpoints

What does this MR do and why?

Optionally allow compressed checkpoints in workflows internal API endpoint to reduce payload size

References

Related to #571913

How to set up and validate locally

See any workflow in the API using accept_compressed=true, you must see the checkpoint as base64 encoded:

http://gdk.test:3000//api/v4/ai/duo_workflows/workflows/XXX/checkpoints?per_page=1&accept_compressed=1

Testing with duo workflow that the current state does not get broken

1. Create a chat and see if all works fine
2. Close and re-open the chat history in vscode and in gitlab UI to see if the history is still there
3. Check in rails console that Ai::DuoWorkflows::Checkpoint.last.checkpoint is correct. (or in database select checkpoint from p_duo_workflows_checkpoints where id=xxx;

Details

• We use base64 and zlib with compression level 6. This compresses > 1MB checkpoint JSONS upto 10x.

bin/rails c

puts "Original: #{File.size('~/checkpoint-3397832.json')} bytes, Compressed: #{Base64.strict_encode64(Zlib::Deflate.deflate(File.read('~/checkpoint-3397832.json'), 6)).bytesize} bytes, Ratio: #{((Base64.strict_encode64(Zlib::Deflate.deflate(File.read('~/checkpoint-3397832.json'),6)).bytesize.to_f / File.size('~/checkpoint-3397832.json')) * 100).round(2)}%"

Original: 1319889 bytes, Compressed: 135544 bytes, Ratio: 10.27%

• Duo workflow service is in python - use the following to prove zlib/base64 is compatible/universal

ruby -r zlib -r base64 -e "puts Base64.strict_encode64(Zlib::Deflate.deflate('[\"Hello\"]', 6))" | python3 -c "import sys, zlib, base64; print(zlib.decompress(base64.b64decode(sys.stdin.read().strip())).decode())"

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.