admin
不忘初心,方得始终
级别: 管理员
只看楼主 | | | 0楼 发表于:2017-02-18 20:45

Openstack负载均衡成员INACTIVE故障的排查问题思路

1.查看异常member的信息,获取租户id和ip,以及负载均衡池的id
[root@node-1 ~](controller)# neutron lb-member-show 8371cd6d-1aa0-41c8-b65a-cf6da2706c72
+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| address            | 192.168.10.30                        |
| admin_state_up     | True                                 |
| id                 | 8371cd6d-1aa0-41c8-b65a-cf6da2706c72 |
| pool_id            | 55122608-20d9-467d-9fb7-f9d312b9ea68 |
| protocol_port      | 80                                   |
| status             | INACTIVE                             |
| status_description |                                      |
| tenant_id          | 175138f1d54b4ec7abbcab331b81fe78     |
| weight             | 10                                   |
+--------------------+--------------------------------------+
2.获取ip对应的port_id
[root@node-1 ~](controller)# neutron port-list | grep 192.168.10.30
| 2510a195-e2e8-4fdc-bdc5-e7f110879215 |                                          | fa:16:3e:ad:a5:2b | {"subnet_id": "f20ffb49-a248-489e-a4df-ba6c8d09030a", "ip_address": "192.168.10.30"}   |
3.根据port_id获取端口对应的device_id,即实例id
[root@node-1 ~](controller)# neutron port-show 2510a195-e2e8-4fdc-bdc5-e7f110879215
+-----------------------+--------------------------------------------------------------------------------------+
| Field                 | Value                                                                                |
+-----------------------+--------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                 |
| allowed_address_pairs |                                                                                      |
| binding:host_id       | node-17.eayun.com                                                                    |
| binding:profile       | {}                                                                                   |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                       |
| binding:vif_type      | ovs                                                                                  |
| binding:vnic_type     | normal                                                                               |
| device_id             | 44b567a8-2547-46b2-8531-8ed61c350ebb                                                 |
| device_owner          | compute:None                                                                         |
| extra_dhcp_opts       |                                                                                      |
| fixed_ips             | {"subnet_id": "f20ffb49-a248-489e-a4df-ba6c8d09030a", "ip_address": "192.168.10.30"} |
| id                    | 2510a195-e2e8-4fdc-bdc5-e7f110879215                                                 |
| mac_address           | fa:16:3e:ad:a5:2b                                                                    |
| name                  |                                                                                      |
| network_id            | 50913226-60ed-4c72-a76f-b4a427fb5fd2                                                 |
| security_groups       | 4b9ed579-70fc-40cc-8c5c-ce52da4010e9                                                 |
| status                | ACTIVE                                                                               |
| tenant_id             | 175138f1d54b4ec7abbcab331b81fe78                                                     |
+-----------------------+--------------------------------------------------------------------------------------+
4.查看实例状态并无异常
[root@node-1 ~](controller)# nova show 44b567a8-2547-46b2-8531-8ed61c350ebb
+--------------------------------------+--------------------------------------------------------------------------------------------------+
| Property                             | Value                                                                                            |
+--------------------------------------+--------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                                                           |
| OS-EXT-AZ:availability_zone          | nova                                                                                             |
| OS-EXT-SRV-ATTR:host                 | node-17.eayun.com                                                                                |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | node-17.eayun.com                                                                                |
| OS-EXT-SRV-ATTR:instance_name        | instance-000098ea                                                                                |
| OS-EXT-STS:power_state               | 1                                                                                                |
| OS-EXT-STS:task_state                | -                                                                                                |
| OS-EXT-STS:vm_state                  | active                                                                                           |
| OS-SRV-USG:launched_at               | 2017-01-01T09:02:21.000000                                                                       |
| OS-SRV-USG:terminated_at             | -                                                                                                |
| accessIPv4                           |                                                                                                  |
| accessIPv6                           |                                                                                                  |
| config_drive                         |                                                                                                  |
| created                              | 2017-01-01T09:02:08Z                                                                             |
| flavor                               | tempISGSXXER (da88418e-5ea8-40f3-8cf6-43f39d1ab4e0)                                              |
| hostId                               | c486e394b52b426a7d410277f40f77f0d2a2f0c95143ae5fedeaa008                                         |
| id                                   | 44b567a8-2547-46b2-8531-8ed61c350ebb                                                             |
| image                                | Attempt to boot from volume - no image supplied                                                  |
| key_name                             | -                                                                                                |
| metadata                             | {}                                                                                               |
| name                                 | zabbix                                                                                           |
| os-extended-volumes:volumes_attached | [{"id": "b90640f2-2fe2-40ab-9c3c-afed5e84ba6f"}, {"id": "03118048-ae07-4c0f-8b46-74aa086a3694"}] |
| progress                             | 0                                                                                                |
| security_groups                      | default                                                                                          |
| status                               | ACTIVE                                                                                           |
| tenant_id                            | 175138f1d54b4ec7abbcab331b81fe78                                                                 |
| updated                              | 2017-01-23T06:53:12Z                                                                             |
| user_id                              | 3431ba5510044591bcd22782aaf97f61                                                                 |
| 拍库 network                         | 192.168.10.30, 175.25.16.181                                                                     |
+--------------------------------------+--------------------------------------------------------------------------------------------------+
5.根据负载均衡池的id,得到vip的id
[root@node-1 ~](controller)# neutron lb-pool-show 55122608-20d9-467d-9fb7-f9d312b9ea68
+------------------------+--------------------------------------------------------------------------------------------------------+
| Field                  | Value                                                                                                  |
+------------------------+--------------------------------------------------------------------------------------------------------+
| admin_state_up         | True                                                                                                   |
| description            |                                                                                                        |
| health_monitors        | 9e4e160d-228b-4c25-a372-e49cfe12bebc                                                                   |
| health_monitors_status | {"monitor_id": "9e4e160d-228b-4c25-a372-e49cfe12bebc", "status": "ACTIVE", "status_description": null} |
| id                     | 55122608-20d9-467d-9fb7-f9d312b9ea68                                                                   |
| lb_method              | ROUND_ROBIN                                                                                            |
| members                | 8371cd6d-1aa0-41c8-b65a-cf6da2706c72                                                                   |
|                        | a87eb6a9-888e-40fa-9f6a-26a568638355                                                                   |
| name                   | https                                                                                                  |
| protocol               | HTTP                                                                                                   |
| provider               | haproxy                                                                                                |
| status                 | ACTIVE                                                                                                 |
| status_description     |                                                                                                        |
| subnet_id              | f20ffb49-a248-489e-a4df-ba6c8d09030a                                                                   |
| tenant_id              | 175138f1d54b4ec7abbcab331b81fe78                                                                       |
| vip_id                 | d99dcb13-fc94-436b-a01e-7c62d03fee8c                                                                   |
+------------------------+--------------------------------------------------------------------------------------------------------+
6.根据vip的id获取vip
[root@node-1 ~](controller)# neutron lb-vip-show d99dcb13-fc94-436b-a01e-7c62d03fee8c
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| address             | 192.168.10.39                        |
| admin_state_up      | True                                 |
| connection_limit    | 65535                                |
| description         |                                      |
| id                  | d99dcb13-fc94-436b-a01e-7c62d03fee8c |
| name                | vip_1486518428864                    |
| pool_id             | 55122608-20d9-467d-9fb7-f9d312b9ea68 |
| port_id             | 031f1818-a9a7-46b1-b18f-e33f2dfce484 |
| protocol            | HTTP                                 |
| protocol_port       | 80                                   |
| session_persistence |                                      |
| status              | ACTIVE                               |
| status_description  |                                      |
| subnet_id           | f20ffb49-a248-489e-a4df-ba6c8d09030a |
| tenant_id           | 175138f1d54b4ec7abbcab331b81fe78     |
+---------------------+--------------------------------------+
7.获取vip对应端口信息里面的节点
[root@node-1 ~](controller)# neutron port-list | grep 192.168.10.39
| 031f1818-a9a7-46b1-b18f-e33f2dfce484 | vip-d99dcb13-fc94-436b-a01e-7c62d03fee8c | fa:16:3e:16:40:da | {"subnet_id": "f20ffb49-a248-489e-a4df-ba6c8d09030a", "ip_address": "192.168.10.39"}   |
[root@node-1 ~](controller)# neutron port-show 031f1818-a9a7-46b1-b18f-e33f2dfce484
+-----------------------+--------------------------------------------------------------------------------------+
| Field                 | Value                                                                                |
+-----------------------+--------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                 |
| allowed_address_pairs |                                                                                      |
| binding:host_id       | node-1.eayun.com                                                                     |
| binding:profile       | {}                                                                                   |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                       |
| binding:vif_type      | ovs                                                                                  |
| binding:vnic_type     | normal                                                                               |
| device_id             | ab739ced-5b27-5633-aae3-5cc19a01260d                                                 |
| device_owner          | neutron:LOADBALANCER                                                                 |
| extra_dhcp_opts       |                                                                                      |
| fixed_ips             | {"subnet_id": "f20ffb49-a248-489e-a4df-ba6c8d09030a", "ip_address": "192.168.10.39"} |
| id                    | 031f1818-a9a7-46b1-b18f-e33f2dfce484                                                 |
| mac_address           | fa:16:3e:16:40:da                                                                    |
| name                  | vip-d99dcb13-fc94-436b-a01e-7c62d03fee8c                                             |
| network_id            | 50913226-60ed-4c72-a76f-b4a427fb5fd2                                                 |
| security_groups       | 4b9ed579-70fc-40cc-8c5c-ce52da4010e9                                                 |
| status                | ACTIVE                                                                               |
| tenant_id             | 175138f1d54b4ec7abbcab331b81fe78                                                     |
+-----------------------+--------------------------------------------------------------------------------------+
8.登录到pool所在节点,根据pool_id找到对应的命名空间,然后telnet 对应端口,发现不通
[root@node-1 ~](controller)# ip netns | grep 55122608-20d9-467d-9fb7-f9d312b9ea68
qlbaas-55122608-20d9-467d-9fb7-f9d312b9ea68
[root@node-1 ~](controller)# ip netns exec qlbaas-55122608-20d9-467d-9fb7-f9d312b9ea68 telnet 192.168.10.30 80
Trying 192.168.10.30...
telnet: connect to address 192.168.10.30: Connection refused