1. 06 Aug, 2012 1 commit
  2. 31 Jul, 2012 1 commit
    • Eric Dumazet's avatar
      ipv4: Restore old dst_free() behavior. · 54764bb6
      Eric Dumazet authored
      commit 404e0a8b (net: ipv4: fix RCU races on dst refcounts) tried
      to solve a race but added a problem at device/fib dismantle time :
      
      We really want to call dst_free() as soon as possible, even if sockets
      still have dst in their cache.
      dst_release() calls in free_fib_info_rcu() are not welcomed.
      
      Root of the problem was that now we also cache output routes (in
      nh_rth_output), we must use call_rcu() instead of call_rcu_bh() in
      rt_free(), because output route lookups are done in process context.
      
      Based on feedback and initial patch from David Miller (adding another
      call_rcu_bh() call in fib, but it appears it was not the right fix)
      
      I left the inet_sk_rx_dst_set() helper and added __rcu attributes
      to nh_rth_output and nh_rth_input to better document what is going on in
      this code.
      Signed-off-by: default avatarEric Dumazet <[email protected]>
      Signed-off-by: default avatarDavid S. Miller <[email protected]>
      54764bb6
  3. 30 Jul, 2012 1 commit
    • Eric Dumazet's avatar
      net: ipv4: fix RCU races on dst refcounts · 404e0a8b
      Eric Dumazet authored
      commit c6cffba4 (ipv4: Fix input route performance regression.)
      added various fatal races with dst refcounts.
      
      crashes happen on tcp workloads if routes are added/deleted at the same
      time.
      
      The dst_free() calls from free_fib_info_rcu() are clearly racy.
      
      We need instead regular dst refcounting (dst_release()) and make
      sure dst_release() is aware of RCU grace periods :
      
      Add DST_RCU_FREE flag so that dst_release() respects an RCU grace period
      before dst destruction for cached dst
      
      Introduce a new inet_sk_rx_dst_set() helper, using atomic_inc_not_zero()
      to make sure we dont increase a zero refcount (On a dst currently
      waiting an rcu grace period before destruction)
      
      rt_cache_route() must take a reference on the new cached route, and
      release it if was not able to install it.
      
      With this patch, my machines survive various benchmarks.
      Signed-off-by: default avatarEric Dumazet <[email protected]>
      Signed-off-by: default avatarDavid S. Miller <[email protected]>
      404e0a8b
  4. 23 Jul, 2012 1 commit
    • David S. Miller's avatar
      ipv4: Prepare for change of rt->rt_iif encoding. · 92101b3b
      David S. Miller authored
      Use inet_iif() consistently, and for TCP record the input interface of
      cached RX dst in inet sock.
      
      rt->rt_iif is going to be encoded differently, so that we can
      legitimately cache input routes in the FIB info more aggressively.
      
      When the input interface is "use SKB device index" the rt->rt_iif will
      be set to zero.
      
      This forces us to move the TCP RX dst cache installation into the ipv4
      specific code, and as well it should since doing the route caching for
      ipv6 is pointless at the moment since it is not inspected in the ipv6
      input paths yet.
      
      Also, remove the unlikely on dst->obsolete, all ipv4 dsts have
      obsolete set to a non-zero value to force invocation of the check
      callback.
      Signed-off-by: default avatarDavid S. Miller <[email protected]>
      92101b3b
  5. 11 Jul, 2012 1 commit
  6. 13 Feb, 2012 1 commit
    • Jiri Benc's avatar
      net: implement IP_RECVTOS for IP_PKTOPTIONS · 4c507d28
      Jiri Benc authored
      Currently, it is not easily possible to get TOS/DSCP value of packets from
      an incoming TCP stream. The mechanism is there, IP_PKTOPTIONS getsockopt
      with IP_RECVTOS set, the same way as incoming TTL can be queried. This is
      not actually implemented for TOS, though.
      
      This patch adds this functionality, both for IPv4 (IP_PKTOPTIONS) and IPv6
      (IPV6_2292PKTOPTIONS). For IPv4, like in the IP_RECVTTL case, the value of
      the TOS field is stored from the other party's ACK.
      
      This is needed for proxies which require DSCP transparency. One such example
      is at http://zph.bratcheda.org/.
      Signed-off-by: Jiri Benc's avatarJiri Benc <[email protected]>
      Signed-off-by: default avatarDavid S. Miller <[email protected]>
      4c507d28
  7. 08 Feb, 2012 1 commit
    • Erich E. Hoover's avatar
      ipv4: Implement IP_UNICAST_IF socket option. · 76e21053
      Erich E. Hoover authored
      The IP_UNICAST_IF feature is needed by the Wine project.  This patch
      implements the feature by setting the outgoing interface in a similar
      fashion to that of IP_MULTICAST_IF.  A separate option is needed to
      handle this feature since the existing options do not provide all of
      the characteristics required by IP_UNICAST_IF, a summary is provided
      below.
      
      SO_BINDTODEVICE:
      * SO_BINDTODEVICE requires administrative privileges, IP_UNICAST_IF
      does not.  From reading some old mailing list articles my
      understanding is that SO_BINDTODEVICE requires administrative
      privileges because it can override the administrator's routing
      settings.
      * The SO_BINDTODEVICE option restricts both outbound and inbound
      traffic, IP_UNICAST_IF only impacts outbound traffic.
      
      IP_PKTINFO:
      * Since IP_PKTINFO and IP_UNICAST_IF are independent options,
      implementing IP_UNICAST_IF with IP_PKTINFO will likely break some
      applications.
      * Implementing IP_UNICAST_IF on top of IP_PKTINFO significantly
      complicates the Wine codebase and reduces the socket performance
      (doing this requires a lot of extra communication between the
      "server" and "user" layers).
      
      bind():
      * bind() does not work on broadcast packets, IP_UNICAST_IF is
      specifically intended to work with broadcast packets.
      * Like SO_BINDTODEVICE, bind() restricts both outbound and inbound
      traffic.
      Signed-off-by: default avatarErich E. Hoover <[email protected]>
      Signed-off-by: default avatarEric Dumazet <[email protected]>
      Signed-off-by: default avatarDavid S. Miller <[email protected]>
      76e21053
  8. 11 Dec, 2011 1 commit
  9. 24 Nov, 2011 1 commit
  10. 08 Aug, 2011 1 commit
  11. 06 May, 2011 1 commit
  12. 28 Apr, 2011 1 commit
    • Eric Dumazet's avatar
      inet: add RCU protection to inet->opt · f6d8bd05
      Eric Dumazet authored
      We lack proper synchronization to manipulate inet->opt ip_options
      
      Problem is ip_make_skb() calls ip_setup_cork() and
      ip_setup_cork() possibly makes a copy of ipc->opt (struct ip_options),
      without any protection against another thread manipulating inet->opt.
      
      Another thread can change inet->opt pointer and free old one under us.
      
      Use RCU to protect inet->opt (changed to inet->inet_opt).
      
      Instead of handling atomic refcounts, just copy ip_options when
      necessary, to avoid cache line dirtying.
      
      We cant insert an rcu_head in struct ip_options since its included in
      skb->cb[], so this patch is large because I had to introduce a new
      ip_options_rcu structure.
      Signed-off-by: default avatarEric Dumazet <[email protected]>
      Cc: Herbert Xu <[email protected]>
      Signed-off-by: default avatarDavid S. Miller <[email protected]>
      f6d8bd05
  13. 01 Mar, 2011 1 commit
  14. 28 Jan, 2011 1 commit
  15. 10 Dec, 2010 1 commit
    • Eric Dumazet's avatar
      net: optimize INET input path further · 68835aba
      Eric Dumazet authored
      Followup of commit b178bb3d (net: reorder struct sock fields)
      
      Optimize INET input path a bit further, by :
      
      1) moving sk_refcnt close to sk_lock.
      
      This reduces number of dirtied cache lines by one on 64bit arches (and
      64 bytes cache line size).
      
      2) moving inet_daddr & inet_rcv_saddr at the beginning of sk
      
      (same cache line than hash / family / bound_dev_if / nulls_node)
      
      This reduces number of accessed cache lines in lookups by one, and dont
      increase size of inet and timewait socks.
      inet and tw sockets now share same place-holder for these fields.
      
      Before patch :
      
      offsetof(struct sock, sk_refcnt) = 0x10
      offsetof(struct sock, sk_lock) = 0x40
      offsetof(struct sock, sk_receive_queue) = 0x60
      offsetof(struct inet_sock, inet_daddr) = 0x270
      offsetof(struct inet_sock, inet_rcv_saddr) = 0x274
      
      After patch :
      
      offsetof(struct sock, sk_refcnt) = 0x44
      offsetof(struct sock, sk_lock) = 0x48
      offsetof(struct sock, sk_receive_queue) = 0x68
      offsetof(struct inet_sock, inet_daddr) = 0x0
      offsetof(struct inet_sock, inet_rcv_saddr) = 0x4
      
      compute_score() (udp or tcp) now use a single cache line per ignored
      item, instead of two.
      Signed-off-by: default avatarEric Dumazet <[email protected]>
      Signed-off-by: default avatarDavid S. Miller <[email protected]>
      68835aba
  16. 12 Nov, 2010 1 commit
  17. 23 Jun, 2010 1 commit
  18. 27 Apr, 2010 2 commits
  19. 16 Apr, 2010 1 commit
    • Tom Herbert's avatar
      rfs: Receive Flow Steering · fec5e652
      Tom Herbert authored
      This patch implements receive flow steering (RFS).  RFS steers
      received packets for layer 3 and 4 processing to the CPU where
      the application for the corresponding flow is running.  RFS is an
      extension of Receive Packet Steering (RPS).
      
      The basic idea of RFS is that when an application calls recvmsg
      (or sendmsg) the application's running CPU is stored in a hash
      table that is indexed by the connection's rxhash which is stored in
      the socket structure.  The rxhash is passed in skb's received on
      the connection from netif_receive_skb.  For each received packet,
      the associated rxhash is used to look up the CPU in the hash table,
      if a valid CPU is set then the packet is steered to that CPU using
      the RPS mechanisms.
      
      The convolution of the simple approach is that it would potentially
      allow OOO packets.  If threads are thrashing around CPUs or multiple
      threads are trying to read from the same sockets, a quickly changing
      CPU value in the hash table could cause rampant OOO packets--
      we consider this a non-starter.
      
      To avoid OOO packets, this solution implements two types of hash
      tables: rps_sock_flow_table and rps_dev_flow_table.
      
      rps_sock_table is a global hash table.  Each entry is just a CPU
      number and it is populated in recvmsg and sendmsg as described above.
      This table contains the "desired" CPUs for flows.
      
      rps_dev_flow_table is specific to each device queue.  Each entry
      contains a CPU and a tail queue counter.  The CPU is the "current"
      CPU for a matching flow.  The tail queue counter holds the value
      of a tail queue counter for the associated CPU's backlog queue at
      the time of last enqueue for a flow matching the entry.
      
      Each backlog queue has a queue head counter which is incremented
      on dequeue, and so a queue tail counter is computed as queue head
      count + queue length.  When a packet is enqueued on a backlog queue,
      the current value of the queue tail counter is saved in the hash
      entry of the rps_dev_flow_table.
      
      And now the trick: when selecting the CPU for RPS (get_rps_cpu)
      the rps_sock_flow table and the rps_dev_flow table for the RX queue
      are consulted.  When the desired CPU for the flow (found in the
      rps_sock_flow table) does not match the current CPU (found in the
      rps_dev_flow table), the current CPU is changed to the desired CPU
      if one of the following is true:
      
      - The current CPU is unset (equal to RPS_NO_CPU)
      - Current CPU is offline
      - The current CPU's queue head counter >= queue tail counter in the
      rps_dev_flow table.  This checks if the queue tail has advanced
      beyond the last packet that was enqueued using this table entry.
      This guarantees that all packets queued using this entry have been
      dequeued, thus preserving in order delivery.
      
      Making each queue have its own rps_dev_flow table has two advantages:
      1) the tail queue counters will be written on each receive, so
      keeping the table local to interrupting CPU s good for locality.  2)
      this allows lockless access to the table-- the CPU number and queue
      tail counter need to be accessed together under mutual exclusion
      from netif_receive_skb, we assume that this is only called from
      device napi_poll which is non-reentrant.
      
      This patch implements RFS for TCP and connected UDP sockets.
      It should be usable for other flow oriented protocols.
      
      There are two configuration parameters for RFS.  The
      "rps_flow_entries" kernel init parameter sets the number of
      entries in the rps_sock_flow_table, the per rxqueue sysfs entry
      "rps_flow_cnt" contains the number of entries in the rps_dev_flow
      table for the rxqueue.  Both are rounded to power of two.
      
      The obvious benefit of RFS (over just RPS) is that it achieves
      CPU locality between the receive processing for a flow and the
      applications processing; this can result in increased performance
      (higher pps, lower latency).
      
      The benefits of RFS are dependent on cache hierarchy, application
      load, and other factors.  On simple benchmarks, we don't necessarily
      see improvement and sometimes see degradation.  However, for more
      complex benchmarks and for applications where cache pressure is
      much higher this technique seems to perform very well.
      
      Below are some benchmark results which show the potential benfit of
      this patch.  The netperf test has 500 instances of netperf TCP_RR
      test with 1 byte req. and resp.  The RPC test is an request/response
      test similar in structure to netperf RR test ith 100 threads on
      each host, but does more work in userspace that netperf.
      
      e1000e on 8 core Intel
         No RFS or RPS		104K tps at 30% CPU
         No RFS (best RPS config):    290K tps at 63% CPU
         RFS				303K tps at 61% CPU
      
      RPC test	tps	CPU%	50/90/99% usec latency	Latency StdDev
        No RFS/RPS	103K	48%	757/900/3185		4472.35
        RPS only:	174K	73%	415/993/2468		491.66
        RFS		223K	73%	379/651/1382		315.61
      Signed-off-by: default avatarTom Herbert <[email protected]>
      Signed-off-by: default avatarEric Dumazet <[email protected]>
      Signed-off-by: default avatarDavid S. Miller <[email protected]>
      fec5e652
  20. 12 Jan, 2010 1 commit
  21. 19 Oct, 2009 1 commit
    • Eric Dumazet's avatar
      inet: rename some inet_sock fields · c720c7e8
      Eric Dumazet authored
      In order to have better cache layouts of struct sock (separate zones
      for rx/tx paths), we need this preliminary patch.
      
      Goal is to transfert fields used at lookup time in the first
      read-mostly cache line (inside struct sock_common) and move sk_refcnt
      to a separate cache line (only written by rx path)
      
      This patch adds inet_ prefix to daddr, rcv_saddr, dport, num, saddr,
      sport and id fields. This allows a future patch to define these
      fields as macros, like sk_refcnt, without name clashes.
      Signed-off-by: default avatarEric Dumazet <[email protected]>
      Signed-off-by: default avatarDavid S. Miller <[email protected]>
      c720c7e8
  22. 15 Jun, 2009 1 commit
  23. 02 Jun, 2009 1 commit
  24. 01 Oct, 2008 4 commits
  25. 17 Jun, 2008 2 commits
  26. 10 Jun, 2008 1 commit
  27. 25 Mar, 2008 1 commit
  28. 22 Mar, 2008 1 commit
  29. 06 Mar, 2008 1 commit
  30. 04 Mar, 2008 1 commit
    • David S. Miller's avatar
      [TCP]: Improve ipv4 established hash function. · 7adc3830
      David S. Miller authored
      If all of the entropy is in the local and foreign addresses,
      but xor'ing together would cancel out that entropy, the
      current hash performs poorly.
      
      Suggested by Cosmin Ratiu:
      
      	Basically, the situation is as follows: There is a client
      	machine and a server machine. Both create 15000 virtual
      	interfaces, open up a socket for each pair of interfaces and
      	do SIP traffic. By profiling I noticed that there is a lot of
      	time spent walking the established hash chains with this
      	particular setup.
      
      	The addresses were distributed like this: client interfaces
      	were 198.18.0.1/16 with increments of 1 and server interfaces
      	were 198.18.128.1/16 with increments of 1. As I said, there
      	were 15000 interfaces. Source and destination ports were 5060
      	for each connection.  So in this case, ports don't matter for
      	hashing purposes, and the bits from the address pairs used
      	cancel each other, meaning there are no differences in the
      	whole lot of pairs, so they all end up in the same hash chain.
      Signed-off-by: default avatarDavid S. Miller <[email protected]>
      7adc3830
  31. 26 Oct, 2007 1 commit
  32. 26 Apr, 2007 1 commit
  33. 29 Sep, 2006 3 commits