• Aaron Lu's avatar
    mem_cgroup: make sure moving_account, move_lock_task and stat_cpu in the same cacheline · e81bf979
    Aaron Lu authored
    The LKP robot found a 27% will-it-scale/page_fault3 performance
    regression regarding commit e27be240("mm: memcg: make sure
    memory.events is uptodate when waking pollers").
    
    What the test does is:
     1 mkstemp() a 128M file on a tmpfs;
     2 start $nr_cpu processes, each to loop the following:
       2.1 mmap() this file in shared write mode;
       2.2 write 0 to this file in a PAGE_SIZE step till the end of the file;
       2.3 unmap() this file and repeat this process.
     3 After 5 minutes, check how many loops they managed to complete, the
       higher the better.
    
    The commit itself looks innocent enough as it merely changed some event
    counting mechanism and this test didn't trigger those events at all.
    Perf shows increased cycles spent on accessing root_mem_cgroup->stat_cpu
    in count_memcg_event_mm()(called by handle_mm_fault()) and in
    __mod_memcg_state() called by page_add_file_rmap().  So it's likely due
    to the changed layout of 'struct mem_cgroup' that either make stat_cpu
    falling into a constantly modifying cacheline or some hot fields stop
    being in the same cacheline.
    
    I verified this by moving memory_events[] back to where it was:
    
    : --- a/include/linux/memcontrol.h
    : +++ b/include/linux/memcontrol.h
    : @@ -205,7 +205,6 @@ struct mem_cgroup {
    :  	int		oom_kill_disable;
    :
    :  	/* memory.events */
    : -	atomic_long_t memory_events[MEMCG_NR_MEMORY_EVENTS];
    :  	struct cgroup_file events_file;
    :
    :  	/* protect arrays of thresholds */
    : @@ -238,6 +237,7 @@ struct mem_cgroup {
    :  	struct mem_cgroup_stat_cpu __percpu *stat_cpu;
    :  	atomic_long_t		stat[MEMCG_NR_STAT];
    :  	atomic_long_t		events[NR_VM_EVENT_ITEMS];
    : +	atomic_long_t memory_events[MEMCG_NR_MEMORY_EVENTS];
    :
    :  	unsigned long		socket_pressure;
    
    And performance restored.
    
    Later investigation found that as long as the following 3 fields
    moving_account, move_lock_task and stat_cpu are in the same cacheline,
    performance will be good.  To avoid future performance surprise by other
    commits changing the layout of 'struct mem_cgroup', this patch makes
    sure the 3 fields stay in the same cacheline.
    
    One concern of this approach is, moving_account and move_lock_task could
    be modified when a process changes memory cgroup while stat_cpu is a
    always read field, it might hurt to place them in the same cacheline.  I
    assume it is rare for a process to change memory cgroup so this should
    be OK.
    
    Link: https://lkml.kernel.org/r/20180528114019.GF9904@yexl-desktop
    Link: http://lkml.kernel.org/r/20180601071115.GA27302@intel.comSigned-off-by: default avatarAaron Lu <aaron.lu@intel.com>
    Reported-by: default avatarkernel test robot <xiaolong.ye@intel.com>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: Michal Hocko <mhocko@kernel.org>
    Cc: Tejun Heo <tj@kernel.org>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    e81bf979
Name
Last commit
Last update
Documentation Loading commit data...
LICENSES Loading commit data...
arch Loading commit data...
block Loading commit data...
certs Loading commit data...
crypto Loading commit data...
drivers Loading commit data...
firmware Loading commit data...
fs Loading commit data...
include Loading commit data...
init Loading commit data...
ipc Loading commit data...
kernel Loading commit data...
lib Loading commit data...
mm Loading commit data...
net Loading commit data...
samples Loading commit data...
scripts Loading commit data...
security Loading commit data...
sound Loading commit data...
tools Loading commit data...
usr Loading commit data...
virt Loading commit data...
.clang-format Loading commit data...
.cocciconfig Loading commit data...
.get_maintainer.ignore Loading commit data...
.gitattributes Loading commit data...
.gitignore Loading commit data...
.mailmap Loading commit data...
COPYING Loading commit data...
CREDITS Loading commit data...
Kbuild Loading commit data...
Kconfig Loading commit data...
MAINTAINERS Loading commit data...
Makefile Loading commit data...
README Loading commit data...