10 min read 路由交换

多路出口带宽代理实现

多路出口带宽代理实现

背景

20条宽带出口实现20个独立IP的代理服务，如用户从出口A访问进来，代理服务也必须是使用出口A去访问被代理的服务。

涉及资源

  • 交换机：华为 S5735S-L48TAS-A *1
  • 路由：Mikrotik RB5009UG+S+（v7.2.3) *1
  • 服务器：CentOS7.9 *1
  • ISP: 电信（PPPoE拨号） * 20

涉及技术点

  • 交换机：VLAN、Trunk、链路聚合等
  • 软路由：SNAT、DNAT、Bridge、策略路由等
  • 代理服务器： nginx（HTTP代理）、dante（Socks5代理）

网段规划

  • 内网网关：172.16.255.1/24
  • 代理服务器：
    • 172.16.255.101/24 <-> pppoe-out01
    • 172.16.255.102/24 <-> pppoe-out02
    • ...
    • 172.16.255.120/24 <-> pppoe-out20

部署要略

交换机设备

根据软规则对交换机进行如下端口规划：

序号 端口号 VLAN 备注
1 GE 0/0/1 101 pppoe-out01
2 GE 0/0/2 102 pppoe-out02
3 GE 0/0/3 103 pppoe-out03
4 GE 0/0/4 104 pppoe-out04
5 GE 0/0/5 105 pppoe-out05
6 GE 0/0/6 106 pppoe-out06
7 GE 0/0/7 107 pppoe-out07
8 GE 0/0/8 108 pppoe-out08
9 GE 0/0/9 109 pppoe-out09
10 GE 0/0/10 110 pppoe-out10
11 GE 0/0/11 111 pppoe-out11
12 GE 0/0/12 112 pppoe-out12
13 GE 0/0/13 113 pppoe-out13
14 GE 0/0/14 114 pppoe-out14
15 GE 0/0/15 115 pppoe-out15
16 GE 0/0/16 116 pppoe-out16
17 GE 0/0/17 117 pppoe-out17
18 GE 0/0/18 118 pppoe-out18
19 GE 0/0/19 119 pppoe-out19
20 GE 0/0/20 120 Pppoe-out20
21 GE 0/0/45 - 聚合链路成员接口
22 GE 0/0/46 - 聚合链路成员接口
23 GE 0/0/47 - 聚合链路成员接口
24 GE 0/0/48 - 聚合链路成员接口

Mikrotik RouterOS设备

创建聚合接口

创建脚本

/interface bonding
add arp=enabled arp-interval=100ms disabled=no mode=802.3ad mtu=1500 name=bond0 primary=none \
    slaves=ether7,ether8,ether5,ether6

image-20221119113653045

基于聚合接口创建vlan子接口

创建脚本

:for no  from=1 to=9 do={  /interface vlan add interface=bond0  mtu=1500 name=("vlan1".$no use-service-tag=no vlan-id=("10".$no) }
:for no  from=10 to=20 do={  /interface vlan add interface=bond0  mtu=1500 name=("vlan1".$no use-service-tag=no vlan-id=("1".$no) }

image-20221119113627703

基于聚合接口创建bridge接口

新增专用bridge接口，脚本如下（部分省略）：

/interface bridge
add admin-mac=E4:8D:8C:1D:24:01 ageing-time=5m arp=enabled arp-timeout=auto auto-mac=no dhcp-snooping=no disabled=no \
    fast-forward=no igmp-snooping=no mtu=auto name=ct-01
add admin-mac=E4:8D:8C:1D:24:02 ageing-time=5m arp=enabled arp-timeout=auto auto-mac=no dhcp-snooping=no disabled=no \
    fast-forward=no igmp-snooping=no mtu=auto name=ct-02
add admin-mac=E4:8D:8C:1D:24:03 ageing-time=5m arp=enabled arp-timeout=auto auto-mac=no dhcp-snooping=no disabled=no \
    fast-forward=no igmp-snooping=no mtu=auto name=ct-03
<...>

image-20221119113604817

将vlan接口添加到对应的bridge接口

脚本如下（部分省略）

/interface bridge port
add auto-isolate=no bpdu-guard=no bridge=ct-01 broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=\
    admit-all horizon=none ingress-filtering=yes interface=vlan101 internal-path-cost=10 learn=auto multicast-router=\
    temporary-query path-cost=10 point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no \
    tag-stacking=no trusted=no unknown-multicast-flood=yes unknown-unicast-flood=yes
<...>

image-20221119113527366

创建pppoe拨号接口

脚本如下（部分省略）

/interface pppoe-client
add add-default-route=yes allow=pap,chap,mschap1,mschap2 default-route-distance=2 dial-on-demand=no disabled=no interface=ct-01\
  name=pppoe-out01 profile=default use-peer-dns=no user=USERNAME
add add-default-route=yes allow=pap,chap,mschap1,mschap2 dial-on-demand=no disabled=no interface=ct-02\
  name=pppoe-out01 profile=default use-peer-dns=no user=USERNAME
<...>

image-20221119113103592

策略路由

每个源IP需要保证固定出口，所以我们需要使用策略路由来实现，大致方法就是根据源IP去决定走哪个出口，实现如下：

脚本如下（部分省略）

  • 创建路由表
:for no  from=1 to=9 do={  /routing table add name=("proxy0".$no) fib }
:for no  from=11 to=20 do={  /routing table add name=("proxy".$no) fib }

玛德，规则软过头了

  • 在Mangle表中的Prerouting链定义源IP并生成20个路由表
/ip/firewall/mangle
  add src-address=172.16.255.101 chain=prerouting action=mark-routing new-routing-mark=proxy01 passthrough=yes
  add src-address=172.16.255.102 chain=prerouting action=mark-routing new-routing-mark=proxy02 passthrough=yes
  add src-address=172.16.255.103 chain=prerouting action=mark-routing new-routing-mark=proxy03 passthrough=yes
  add src-address=172.16.255.104 chain=prerouting action=mark-routing new-routing-mark=proxy04 passthrough=yes
  add src-address=172.16.255.105 chain=prerouting action=mark-routing new-routing-mark=proxy05 passthrough=yes
  add src-address=172.16.255.106 chain=prerouting action=mark-routing new-routing-mark=proxy06 passthrough=yes
  add src-address=172.16.255.107 chain=prerouting action=mark-routing new-routing-mark=proxy07 passthrough=yes
  add src-address=172.16.255.108 chain=prerouting action=mark-routing new-routing-mark=proxy08 passthrough=yes
  add src-address=172.16.255.109 chain=prerouting action=mark-routing new-routing-mark=proxy09 passthrough=yes
  add src-address=172.16.255.110 chain=prerouting action=mark-routing new-routing-mark=proxy10 passthrough=yes
  add src-address=172.16.255.111 chain=prerouting action=mark-routing new-routing-mark=proxy11 passthrough=yes
  add src-address=172.16.255.112 chain=prerouting action=mark-routing new-routing-mark=proxy12 passthrough=yes
  add src-address=172.16.255.113 chain=prerouting action=mark-routing new-routing-mark=proxy13 passthrough=yes
  add src-address=172.16.255.114 chain=prerouting action=mark-routing new-routing-mark=proxy14 passthrough=yes
  add src-address=172.16.255.115 chain=prerouting action=mark-routing new-routing-mark=proxy15 passthrough=yes
  add src-address=172.16.255.116 chain=prerouting action=mark-routing new-routing-mark=proxy16 passthrough=yes
  add src-address=172.16.255.117 chain=prerouting action=mark-routing new-routing-mark=proxy17 passthrough=yes
  add src-address=172.16.255.118 chain=prerouting action=mark-routing new-routing-mark=proxy18 passthrough=yes
  add src-address=172.16.255.119 chain=prerouting action=mark-routing new-routing-mark=proxy19 passthrough=yes
  add src-address=172.16.255.120 chain=prerouting action=mark-routing new-routing-mark=proxy20 passthrough=yes
  • 根据创建好的路由表，添加路由
:for no  from=1 to=9 do={  /ip/route add dst-address=0.0.0.0/0 gateway=("pppoe-out0".$no) routing-table=("proxy0".$no) }
:for no  from=10 to=20 do={  /ip/route add dst-address=0.0.0.0/0 gateway=("pppoe-out".$no) routing-table=("proxy".$no) }

规则下次适度软下就好了。

  • 创建SNAT规则
:for no  from=1 to=9 do={  /ip firewall nat add action=masquerade chain=srcnat out-interface=("pppoe-out0".$no) src-address=("172.16.255.10".$no) }
:for no  from=10 to=20 do={ /ip firewall nat add action=masquerade chain=srcnat out-interface=("pppoe-out".$no) src-address=("172.16.255.1".$no) }

image-20221119114937832

  • 创建DNAT规则
/ip firewall nat
add chain=dstnat action=dst-nat to-addresses=172.16.255.101 to-ports=3014 protocol=tcp src-address-list=src-access-proxy in-interface=pppoe-out01 dst-port=3014 log=no
<...>

add chain=dstnat action=dst-nat to-addresses=172.16.255.101 to-ports=3015 protocol=tcp src-address-list=src-access-proxy in-interface=pppoe-out07 dst-port=3015 log=no
<...>

因未配置用户名密码且客户都是固定IP，所以定义了一个源地址访问地址列表：src-access-proxy

代理服务器

多IP配置

往`/etc/sysconfig/network-scripts/ifcfg-eth0文件中加入以下内容：

IPADDR1=172.16.255.102
PREFIX1=24
IPADDR2=172.16.255.103
PREFIX2=24
IPADDR3=172.16.255.104
PREFIX3=24
IPADDR4=172.16.255.105
PREFIX4=24
IPADDR5=172.16.255.106
PREFIX5=24
IPADDR6=172.16.255.107
PREFIX6=24
IPADDR7=172.16.255.108
PREFIX7=24
IPADDR8=172.16.255.109
PREFIX8=24
IPADDR9=172.16.255.110
PREFIX9=24
IPADDR10=172.16.255.111
PREFIX10=24
IPADDR11=172.16.255.112
PREFIX11=24
IPADDR12=172.16.255.113
PREFIX12=24
IPADDR13=172.16.255.114
PREFIX13=24
IPADDR14=172.16.255.115
PREFIX14=24
IPADDR15=172.16.255.116
PREFIX15=24
IPADDR16=172.16.255.117
PREFIX16=24
IPADDR17=172.16.255.118
PREFIX17=24
IPADDR18=172.16.255.119
PREFIX18=24
IPADDR19=172.16.255.120
PREFIX19=24

快速生效

# nmcli conn reload 
# nmcli conn up eth0

image-20221119120744880

socks5代理

批量生成配置文件、systemd服务

#!/usr/bin/env bash

prefix=172.16.255
port=3014

for no in {101..120}
do
cat  > /etc/sockd_$no.conf<<EOF
logoutput: syslog
user.privileged: root
user.unprivileged: nobody

internal: $prefix.$no port=$port
external: $prefix.$no

socksmethod: none
clientmethod: none

client pass {
    from: 0.0.0.0/0 to: 0.0.0.0/0
    log: connect disconnect error
}

socks pass {
    from: 0.0.0.0/0 to: 0.0.0.0/0
    log: error
}
EOF

cat  > /usr/lib/systemd/system/sockd_$no.service<<EOF
[Unit]
Description=Dante SOCKS proxy
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/sbin/sockd -f /etc/sockd_$no.conf
StandardOutput=syslog
StandardError=syslog
LimitNPROC=infinity
LimitNOFILE=10240
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

systemctl start sockd_$no.service
systemctl enable sockd_$no.service
done

查看端口监听状态

# ss -tln |grep 3014
LISTEN     0      511    172.16.255.120:3014                     *:*
LISTEN     0      511    172.16.255.119:3014                     *:*
LISTEN     0      511    172.16.255.118:3014                     *:*
LISTEN     0      511    172.16.255.117:3014                     *:*
LISTEN     0      511    172.16.255.116:3014                     *:*
LISTEN     0      511    172.16.255.115:3014                     *:*
LISTEN     0      511    172.16.255.114:3014                     *:*
LISTEN     0      511    172.16.255.113:3014                     *:*
LISTEN     0      511    172.16.255.112:3014                     *:*
LISTEN     0      511    172.16.255.111:3014                     *:*
LISTEN     0      511    172.16.255.110:3014                     *:*
LISTEN     0      511    172.16.255.109:3014                     *:*
LISTEN     0      511    172.16.255.108:3014                     *:*
LISTEN     0      511    172.16.255.107:3014                     *:*
LISTEN     0      511    172.16.255.106:3014                     *:*
LISTEN     0      511    172.16.255.105:3014                     *:*
LISTEN     0      511    172.16.255.104:3014                     *:*
LISTEN     0      511    172.16.255.103:3014                     *:*
LISTEN     0      511    172.16.255.102:3014                     *:*
LISTEN     0      511    172.16.255.101:3014                     *:*

HTTP代理

http代理第一闪现的是squid，经过测试squid在单节点多IP的场景不受控制，遂转入到nginx，使用以下脚本快速生成配置文件

prefix=172.16.255

for no in {101..120}
do
cat >/opt/nginx/conf/conf.d/$no.conf<<EOF
server {
    listen   $prefix.$no:3015;

    # forward proxy for CONNECT request
    proxy_connect;
    proxy_connect_allow            all;
    proxy_connect_connect_timeout  30s;
    proxy_connect_read_timeout     90s;
    proxy_connect_send_timeout     90s;
    
    proxy_connect_bind     $prefix.$no;
    
    location / {
        proxy_pass \$scheme://\$http_host\$request_uri;
        proxy_bind $prefix.$no;
    }

    access_log /var/log/nginx/${no}_access.log combined buffer=64k flush=1m;
}
EOF
done

测试

测试脚本

#!/usr/bin/env bash


proxy_ips=(
   IP1
   IP2
   ...
   IP20
)

url=${1:-myip.ipip.net}

echo "[+] socks5 proxy testing"
for ip in ${proxy_ips[@]}
do
    echo "[-]$ip testing result: $(curl -s /dev/null --connect-timeout 3 -x socks5://$ip:3014 $url)"
done

echo "[+] http proxy testing"
for ip in ${proxy_ips[@]}
do
    echo "[-]$ip testing result: $(curl -s /dev/null --connect-timeout 3 -x socks5://$ip:3015 $url)"
done

测试结果：代理IP的返回来的IP一致。
ip138

后续

DDNS

由于是当前是使用IP，可以使用域名来绑定出口。

问题答疑

1、代理服务器重启后，有几个socks 地址代理失败，经过查看，失败是因为系统未监听该端口，查看日志，内存资源分配失败导致，当前虚拟机内存1G。加资源管上。

You might also like...

Feb
24
L2TP/IPsec VPN快速部署

L2TP/IPsec VPN快速部署

简介 运维部署图一个快，于是本实例使用容器直接跑一个L2TP/IPSec VPN 服务。直接将家中的路由(Mikrotik hAP AC2)与目标网络打通。步骤如下： 部署过程 拉取镜像 docker pull teddysun/l2tp:latest 创建配置文件
1 min read
Dec
08
离线环境下妙用bin安装包

离线环境下妙用bin安装包

2 min read
Oct
16
nginx性能调优要点

nginx性能调优要点

1、提高worker连接限制 events { worker_connections 100000; } 提示：缺省为1024 2、优化worker数量 worker_processes auto; 3、配置UpStream连接复用 upstream demo1 { server 1.1.
5 min read
Sep
11
趣玩 OSS + docker registry + consul-template 组合

趣玩 OSS + docker registry + consul-template 组合

把用了6年多的ECS操作系统（Deiban8）升级了一番，百无聊赖，找点乐子来打发下。于是想到工作上时常拉取容器镜像那个龟速，就折腾它啦，顺便掺和些奇技淫巧，不亦乐乎。 docker-registry docker-registry是docker官方出品的轻量级别服务，就咱们这种穷人，ECS资源毕竟有限，要是资源有足，咱也不折腾了，直接部署个Habor、Nexus就完事了。 同时考虑到本地存储也是捉襟见肘，直接将容器镜像存储在阿里的OSS上面，docker-registry
5 min read
Aug
27
istio改写server响应头部

istio改写server响应头部

istio 作为web服务器，默认会有个server: istio-envoy响应头部，如下： # curl -I localhost HTTP/1.1 404 Not Found date: Sat, 27 Aug 2022 09:
1 min read