Skip to content

feat: mask sensitive data in the DLE logs output

Artyom Kartasov requested to merge filter-logs-output into master

Description

Filters all DLE logs and masks sensitive data

The change in the performance was not measured. I hope that overhead of the filtering for such an amount of logs will not be significant.

If some problems appear, performance should be measured.

Related issue

Examples

2023/03/06 20:47:26 loaders.go:32: [DEBUG]  Config loaded &{{********  2345 true} {{6000 6010} postgresai/extended-postgres:14 false false map[shm-size:128mb]} {120 localhost} {  false} {{postgres postgres} postgres true {false https://postgres.ai/api/general}} {{26 * * * *} [] map[logicalDump:{ map[containerConfig:map[shm-size:128mb] databases:map[test:<nil>] dockerImage:postgresai/extended-postgres:14 dumpLocation:/tmp/dumps parallelJobs:1 source:map[connection:map[dbname:postgres host:172.17.0.2 ******** port:5432 username:postgres] type:local]]} logicalRestore:{ map[configs:map[maintenance_work_mem:500MB shared_buffers:1GB shared_preload_libraries:pg_stat_statements, pg_stat_kcache, auto_explain, logerrors work_mem:100MB] containerConfig:map[shm-size:128mb] databases:map[test:<nil>] dockerImage:postgresai/extended-postgres:14 dumpLocation:/tmp/dumps forceInit:true parallelJobs:1 queryPreprocessing:map[inline: maxParallelWorkers:2 queryPath:]]} logicalSnapshot:{ map[configs:map[maintenance_work_mem:500MB shared_buffers:1GB shared_preload_libraries:pg_stat_statements, pg_stat_kcache, auto_explain, logerrors work_mem:100MB] dataPatching:map[containerConfig:map[shm-size:128mb] dockerImage:postgresai/extended-postgres:14 queryPreprocessing:map[inline: maxParallelWorkers:2 queryPath:]] preprocessingScript:]}]} {map[]} {0 0 0s 0} {/var/lib/dblab/pools clones data sockets observer _pre } {true postgresai/ce-ui:latest 127.0.0.1 2346} {7}}


2023/03/06 20:47:52 main.go:344: [INFO]   Reloading configuration
2023/03/06 20:47:52 loaders.go:32: [DEBUG]  Config loaded &{{********  2345 true} {{6000 6010} postgresai/extended-postgres:14 false false map[shm-size:128mb]} {120 localhost} { ******** false} {{postgres postgres} postgres true {false https://postgres.ai/api/general}} {{26 * * * *} [] map[logicalDump:{ map[containerConfig:map[shm-size:128mb] databases:map[test:<nil>] dockerImage:postgresai/extended-postgres:14 dumpLocation:/tmp/dumps parallelJobs:1 source:map[connection:map[dbname:postgres host:172.17.0.2 ******** port:5432 username:postgres] type:local]]} logicalRestore:{ map[configs:map[maintenance_work_mem:500MB shared_buffers:1GB shared_preload_libraries:pg_stat_statements, pg_stat_kcache, auto_explain, logerrors work_mem:100MB] containerConfig:map[shm-size:128mb] databases:map[test:<nil>] dockerImage:postgresai/extended-postgres:14 dumpLocation:/tmp/dumps forceInit:true parallelJobs:1 queryPreprocessing:map[inline: maxParallelWorkers:2 queryPath:]]} logicalSnapshot:{ map[configs:map[maintenance_work_mem:500MB shared_buffers:1GB shared_preload_libraries:pg_stat_statements, pg_stat_kcache, auto_explain, logerrors work_mem:100MB] dataPatching:map[containerConfig:map[shm-size:128mb] dockerImage:postgresai/extended-postgres:14 queryPreprocessing:map[inline: maxParallelWorkers:2 queryPath:]] preprocessingScript:]}]} {map[]} {0 0 0s 0} {/var/lib/dblab/pools clones data sockets observer _pre } {true postgresai/ce-ui:latest 127.0.0.1 2346} {7}}

Checklist

  • MR description has been reviewed
  • MR changes are functionally tested
  • MR does NOT have API/CLI changes OR there are API/CLI changes and they have been reviewed & DOCS ARE ADJUSTED (reference doc, etc)
  • MR does NOT have UI changes OR there are UI changes and they have been reviewed & UX IS REVIEWED

Merge request reports