Skip to content

Unify logs rendering code path (+ improve performance)

Igor Ponomarev requested to merge job-logs-unify-render-paths into master

Right now there are essentially 2 different code paths that render logs on the job details page. First is inside the django templates and second one is inside javascript incremental logs.

However, this is suboptimal because it requires maintaining two separate code paths. Instead merge it in to one code path inside javascript.

For initial logs already present on the server use the json_script filter provided by Django. It will convert the logs in to the JSON object that can later be parsed by the javascript.

Move the action link list rendering to javascript meaning the _pipeline_actions.html template is no longer needed.

This improves performance because there will be less load on the server to render templates and reduces the required data transfer because logs will be compressed in to compact JSON instead of HTML.

Apache Bench 20 seconds:

Before:

Benchmarking localhost (be patient)
Finished 35 requests


Server Software:        gunicorn
Server Hostname:        localhost
Server Port:            37899

Document Path:          /scheduler/job/4801718
Document Length:        1409743 bytes

Concurrency Level:      1
Time taken for tests:   20.020 seconds
Complete requests:      35
Failed requests:        0
Total transferred:      49356615 bytes
HTML transferred:       49341005 bytes
Requests per second:    1.75 [#/sec] (mean)
Time per request:       571.989 [ms] (mean)
Time per request:       571.989 [ms] (mean, across all concurrent requests)
Transfer rate:          2407.63 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   533  572  34.7    577     739
Waiting:      532  571  34.7    576     739
Total:        533  572  34.7    577     739

Percentage of the requests served within a certain time (ms)
  50%    576
  66%    581
  75%    582
  80%    584
  90%    587
  95%    602
  98%    739
  99%    739
 100%    739 (longest request)

After:

Benchmarking localhost (be patient)
Finished 49 requests


Server Software:        gunicorn
Server Hostname:        localhost
Server Port:            37899

Document Path:          /scheduler/job/4801718
Document Length:        1152380 bytes

Concurrency Level:      1
Time taken for tests:   20.373 seconds
Complete requests:      49
Failed requests:        0
Total transferred:      56488474 bytes
HTML transferred:       56466620 bytes
Requests per second:    2.41 [#/sec] (mean)
Time per request:       415.783 [ms] (mean)
Time per request:       415.783 [ms] (mean, across all concurrent requests)
Transfer rate:          2707.68 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   397  416  42.0    402     610
Waiting:      396  415  42.0    402     610
Total:        397  416  42.0    402     610

Percentage of the requests served within a certain time (ms)
  50%    402
  66%    405
  75%    412
  80%    424
  90%    433
  95%    444
  98%    610
  99%    610
 100%    610 (longest request)

As you can see after the changes the requests are on average 150 milliseconds faster and 257 kB smaller.

Edited by Igor Ponomarev

Merge request reports