Adding SO_REUSEPORT support to illumos

There was ongoing work for this here. but was not integrated into illumos-gate. SO_REUSEPORT seemed like a nice to have in illumos. So I started taking the code on this branch and syncing it against current illumos-joyent, and ended up having working version that needs testing. My first test apart from the testcases already provided in the patch was to compile unbound with -DSO_REUSEPORT to start testing it, in real workloads. So far so good, in illumos-omnios with SO_REUSEPORT support I see this:

neirac@bloody-dev:/etc/opt/ooce/unbound$ netstat -f inet -an | egrep -e '.53'
      *.53                                Idle
      *.53                                Idle
      *.53                                Idle
      *.53                                Idle
      *.53                 *.*                 0      0 128000      0 LISTEN
      *.53                 *.*                 0      0 128000      0 LISTEN
      *.53                 *.*                 0      0 128000      0 LISTEN
      *.53                 *.*                 0      0 128000      0 LISTEN

Meanwhile in a vm with illumos-omnios without SO_REUSEPORT

neirac@omnios-lts:/etc/opt/ooce/unbound$ netstat -na -f inet | egrep -e '.53'
      *.53                                Idle
      *.53                 *.*                 0      0 128000      0 LISTEN

This is expected as with SO_REUSEPORT there will be one socket per cpu. In the case of unbound (1 TCP socket and 1 UDP socket). with so-reuseport: yes we see 4 UDP and 4 UDP sockets.

References