OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#4490 — route reflector IPv4
Scheduled Maintenance Report for Network & Infrastructure
Completed
We will simplify the configurations on the
network by setting 3 routers reflectors
that will take and centralize all the BGP
announcements of all backbone routers,
recalculate the best route and then redistribute
the BGP table on all routers.

We should gain in BGP performance and in reliability
on the optic fibre cuts.

Update(s):

Date: 2010-09-25 16:40:53 UTC
We have always the same error messages on the 3 route collectors :

Sep 25 13:51:50 UTC: %PLATFORM-3-ELEMENT_CRITICAL: R0/0: smand:
ESP/0: Committed Memory value 323% exceeds critical level 320%
Sep 25 14:02:00 UTC: %PLATFORM-3-ELEMENT_CRITICAL: R0/0: smand:
ESP/0: Committed Memory value 323% exceeds critical level 320%
Sep 25 14:12:10 UTC: %PLATFORM-3-ELEMENT_CRITICAL: R0/0: smand:
ESP/0: Committed Memory value 323% exceeds critical level 320%
Sep 25 14:22:20 UTC: %PLATFORM-3-ELEMENT_CRITICAL: R0/0: smand:
ESP/0: Committed Memory value 323% exceeds critical level 320%
Sep 25 14:32:30 UTC: %PLATFORM-3-ELEMENT_CRITICAL: R0/0: smand:
ESP/0: Committed Memory value 323% exceeds critical level 320%
Sep 25 14:42:40 UTC: %PLATFORM-3-ELEMENT_CRITICAL: R0/0: smand:
ESP/0: Committed Memory value 323% exceeds critical level 320%
Sep 25 14:52:50 UTC: %PLATFORM-3-ELEMENT_CRITICAL: R0/0: smand:
ESP/0: Committed Memory value 323% exceeds critical level 320%
Sep 25 15:03:00 UTC: %PLATFORM-3-ELEMENT_CRITICAL: R0/0: smand:
ESP/0: Committed Memory value 323% exceeds critical level 320%
Sep 25 15:13:10 UTC: %PLATFORM-3-ELEMENT_CRITICAL: R0/0: smand:
ESP/0: Committed Memory value 323% exceeds critical level 320%

It may be a bug listed by cisco CSCtd83822

CSCtd83822

Increasing memory usage of `reflector.sh' and `droputil.sh' process
may occur on the ASR 1000 Router Series.

Workaround: None

https://supportforums.cisco.com/thread/2040810?decorator=print&displayFullThread=true
http://www.cisco.com/en/US/docs/ios/ios_xe/2/release/notes/rnasr21.html


Date: 2010-09-21 23:43:30 UTC
Done.

Date: 2010-09-21 23:43:19 UTC
We are cleaning all useless BGP sessions.



Date: 2010-09-19 19:49:28 UTC
Work on the reflector IPv4 was completed.
We will start work on IPv6.



Date: 2010-09-09 08:19:48 UTC
done

Date: 2010-09-09 08:19:34 UTC
rbx-99 cloud computing

Date: 2010-09-09 08:19:19 UTC
vss-3 done.

Date: 2010-09-09 08:11:27 UTC
vss-1 done.

Date: 2010-09-09 08:11:09 UTC
vss-2 done.

Date: 2010-09-09 08:10:46 UTC
done.

We will remove the same routes from BGP.



Date: 2010-09-09 08:09:48 UTC
We will insert the OSPF routes in the network.

Date: 2010-09-07 09:27:25 UTC
Right now , all the backbone runs in the
\"route reflector\" configuration.

Still to be done :

-The BGP rewriting to OPSF at the level of certain interfaces.
- put IPv6 on the \"route reflector\" configuration.


Date: 2010-09-07 09:24:17 UTC
We have finished the working for the changes
to the route refelectors of ip failover p19 ,roubaix1
and dc1/gsw at the level of the housing.

http://status.ovh.net/?do=details&id=502
http://status.ovh.net/?do=details&id=503
http://status.ovh.net/?do=details&id=501



Date: 2010-09-07 07:36:19 UTC
We are going to cut the the m1/m2 router announcements from roubaix1 to rbx-1.
The three route refelctors will retake the announcements.

Date: 2010-09-07 07:30:02 UTC
th2 done
gsw-1 done
gsw-2 done
rbx-1 done
rbx-2 done

Date: 2010-09-07 07:28:54 UTC
done.

th2

Date: 2010-09-07 07:28:16 UTC
We continue with the th1.

Date: 2010-09-07 07:27:37 UTC
All m1/m2 are in the 3 routes reflectors.

Date: 2010-09-07 07:26:48 UTC
We are setting the third rf.

Date: 2010-09-07 07:26:02 UTC
We are going to change the rf-1 on the new hardware.

Date: 2010-09-07 07:23:44 UTC
done.

bru-1
done

Date: 2010-09-07 07:22:35 UTC
We start fra-5. 175Mb of available RAM.

Date: 2010-08-26 13:22:30 UTC
Aug 26 15:03:41 20G.ldn-1-6k.routers.ovh.net 73683: Aug 26 14:03:17 GMT: %FIB-3-NOMEM: Malloc Failure, disabling DCEF

Date: 2010-08-26 13:22:13 UTC
ldn-1-6k#sh mem stat
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 44B199D0 927852080 879716120 48135960 33992048 26522704
I/O 8000000 67108864 11968016 55140848 50129760 54998488



Date: 2010-08-26 13:21:19 UTC
ams-1-6k

crash

Date: 2010-08-26 11:30:20 UTC
p19-7

Date: 2010-08-26 11:26:28 UTC
done
P19-57
done
p19-2

Date: 2010-08-26 11:24:42 UTC
We switch to p19-52.

Date: 2010-08-25 08:14:44 UTC
RF-1 configuration and RF2 are the same.

Now we can continue the development of the route reflector
from tomorrow on the more complicated routers in terms of
configuration. A lot of things to check.



Date: 2010-08-25 08:12:45 UTC
Aug 24 21:53:44 UTC: %PLATFORM-4-ELEMENT_WARNING: R0/0: smand: ESP/0: Committed Memory value 311% exceeds warning level 310%


Date: 2010-08-25 08:12:32 UTC
We have implemented RF-2-a1. The sessions are being
mounted with all the backbone routers.



Date: 2010-08-25 08:11:20 UTC
We have received the ASR 1000.

rf-2-a1#sh mem stat
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 2C085008 1821505244 160192080 1661313164 1660142120 1658947932
lsmpi_io 98BE21D0 6295088 6294120 968 968 968

2Go de RAM ... !?

http://www.cisco.com/en/US/products/ps9343/prod_models_comparison.html

We have said 4Go !? WTF ???

Date: 2010-08-23 19:30:19 UTC
mar-1-6k done
mad-1-6k done

That's all for today. We will check if everything
is working.

rf-1-6k#sh mem stat
Head Total(b) Used(b) Free(b) Lowest(b)
Largest(b)
Processor 468A6CD0 896865072 846133672 50731400 9822828
10606192
I/O 8000000 67108864 21958860 45150004 43552040
44200284


50Mb free. We progress in the simplifying of the BGP :)

Date: 2010-08-23 19:27:06 UTC
lyo-1-6k done
we change to mar-1-6k

Date: 2010-08-23 19:26:26 UTC
TIX announces also 194.42.48.0/24 in public.
We will contact it too.

Date: 2010-08-23 19:25:32 UTC
No problem

We change to zur-1-6k

Date: 2010-08-23 19:25:03 UTC
No problem.

We change to mil-1-6k

Date: 2010-08-23 19:24:11 UTC
No problem.

We change to var-1-6k

Date: 2010-08-23 19:22:57 UTC
We will add pra-1-6k in the collector

Date: 2010-08-23 19:22:17 UTC
VIX works on 193.203.0.0/24 which is a private network.
For an unknow reason VIX announces this network on Internet.
The trafic was redirected. We have filtered the announcements
in order to avoid having 193.203.0.0/24 in the BGP

Date: 2010-08-23 19:16:30 UTC
We will put the configuration on vie-1-6k

Date: 2010-08-23 18:03:54 UTC
We took a good old 6509 in BXL and we succeeded to mount
all the BGP sessions of all the routers.

It's due to the limit of the available memory : 1 Gb

rf-1-6k#sh mem stat
Head Total(b) Used(b) Free(b) Lowest(b)
Largest(b)
Processor 468A6CD0 896865072 850757364 46107708 9822828
10606192
I/O 8000000 67108864 21991872 45116992 44793952
44512252

We need 850Mb to take all peer informations. 46 Mb are left :)

The router does only that. We are waiting for the ASR 1000

1 4 16276 4876914 117339 30402640 23 0 14:12:37
349852
2 4 16276 16905742 117333 30402640 0 0 14:06:13
431448
3 4 16276 18926841 117333 30402640 68 0 14:06:55
432619
4 4 16276 9158 140329 30402640 0 0 14:21:59
21415
6 4 16276 4694 140309 30402640 0 0 14:21:37
13029
8 4 16276 58 139558 30402640 0 0 00:55:08
3
1 4 16276 24580 116550 30402640 0 0 00:45:29
90369
3 4 16276 16063720 117316 30402640 0 0 13:49:16
432643
4 4 16276 4607205 117315 30402640 31 0 13:48:20
431427
5 4 16276 446715 117340 30402640 0 0 14:13:59
432622
6 4 16276 9738 140281 30402640 0 0 14:23:32
14880
7 4 16276 1320 119767 30402640 0 0 14:19:55
1285
8 4 16276 7998676 117334 30402640 0 0 14:07:23
432647
9 4 16276 6852706 117340 30402640 58 0 14:13:30
432622
0 4 16276 374696 117345 30402640 1 0 14:19:01
432622
1 4 16276 6332102 117315 30402640 29 0 13:48:46
423315
0 4 16276 21704 116549 30402640 0 0 00:44:18
97863
1 4 16276 16261554 117335 30402640 57 0 14:08:12
432621
4 4 16276 12933397 117314 30402640 0 0 13:47:49
430017
5 4 16276 5040207 117354 30402640 16 0 14:27:05
432612
2 4 16276 4328253 117361 30402734 10 0 14:24:32
432603
3 4 16276 16765697 117344 30402734 73 0 14:07:43
432621
4 4 16276 444294 117351 30402734 0 0 14:14:32
432626
5 4 16276 17336635 117324 30402734 11 0 13:47:09
431202
6 4 16276 18485 116558 30402734 0 0 00:44:05
90377
7 4 16276 20318 116557 30402734 0 0 00:43:56
0
8 4 16276 6001 140301 30402734 0 0 14:20:48
7499

oh yeah ! :)

rf-1-6k#sh ip route summary
IP routing table name is Default-IP-Routing-Table(0)
IP routing table maximum-paths is 32
Route Source Networks Subnets Overhead Memory (bytes)
connected 0 2 144 288
static 0 0 0 0
ospf 16276 13 239 36144 38332
Intra-area: 245 Inter-area: 6 External-1: 1 External-2: 0
NSSA External-1: 0 NSSA External-2: 0
bgp 16276 138410 294176 31146192 62368012
External: 0 Internal: 432586 Local: 0
internal 5489 12009932
Total 143912 294417 31182480 74416564
Removing Queue Size 0

Date: 2010-08-23 17:55:29 UTC
2010 Aug 19 20:08:43 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 5 Up
2010 Aug 19 20:11:15 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 2 Up
2010 Aug 19 20:11:41 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 4 Up
2010 Aug 19 20:12:58 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 6 Up
2010 Aug 19 20:13:32 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 6 Up
2010 Aug 19 20:14:17 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 8 Up
2010 Aug 19 20:15:23 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 7 Up
2010 Aug 19 20:17:18 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 0 Up
2010 Aug 19 20:18:01 rbx-97-n7-routing %BGP-3-ATTRID_OP: bgp-16276
[7084] Failed to find attribute ID
2010 Aug 19 20:18:01 rbx-97-n7-routing %BGP-3-NOMEMORY: bgp-16276
[7084] Could not allocate Attr entry, attr id
2010 Aug 19 20:18:01 rbx-97-n7-routing %BGP-2-PEERSHALTED: bgp-16276
[7084] BGP all internal peers shutdown due to no memory condition
(Error in sof
t reconfig processing of prefix)
2010 Aug 19 20:18:01 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 4 Down - out of resource error
2010 Aug 19 20:18:01 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 6 Down - out of resource error
2010 Aug 19 20:18:01 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 6 Down - out of resource error
2010 Aug 19 20:18:01 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 7 Down - out of resource error
2010 Aug 19 20:18:01 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 10 Down - out of resource error
2010 Aug 19 20:18:01 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 35 Down - out of resource error
2010 Aug 19 20:18:01 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 92 Down - out of resource error
2010 Aug 19 20:18:01 rbx-97-n7-routing %BGP-5-ADJCHANGE: bgp-16276
[7084] (default) neighbor 98 Down - out of resource error

# sh proc mem shared | i \"urib \"
Component Shared Memory Size Used
Available Ref
Address (kbytes) (kbytes)
(kbytes) Count
urib 0X52DD0000 256000* 21974
234026 16

Only 256Mb of RAM. With the licence that unbinds the
XL functionnalities

Feature Ins Lic Status Expiry Date Comments
Count
--------------------------------------------------------------------------------
SCALABLE_SERVICES_PKG Yes - In use Never -
TRANSPORT_SERVICES_PKG No - Unused -
LAN_ADVANCED_SERVICES_PKG Yes - In use Never -
LAN_ENTERPRISE_SERVICES_PKG Yes - In use Never -
--------------------------------------------------------------------------------

and the context to the max :

limit-resource u4route-mem minimum 250 maximum 250

rbx-97-n7# conf t
Enter configuration commands, one per line. End with CNTL/Z.
rbx-97-n7(config)# vdc routing id 2
rbx-97-n7(config-vdc)# limit-resource u4route-mem minimum ?
Minimum route memory value

rbx-97-n7(config-vdc)# limit-resource u4route-mem minimum 250 maximum
?
Maximum route memory value

rbx-97-n7(config-vdc)# limit-resource u4route-mem minimum 250 maximum
^C
rbx-97-n7(config-vdc)#

Only 256 Mb of RAM possible to take all the routes.
That's not what the cisco.com site says (2Gb of
RAM on the XL card and 4Gb on the sup ) but well ... Marketing.

So the conclusion is simple : Nexus 7000 is not usable at OVH.


Date: 2010-08-23 17:44:56 UTC
The Nexus 7000 does not allows to do it.

It has only 256 Mb of RAM in one context and cannot take several
BGP full route sessions to recalculate them. It crashes.

Date: 2010-08-23 17:37:04 UTC
We should receive the 3 ASR 1000 by the end of August.

In the meanntime, we test this functionnality on the nexus 7000.
Posted Aug 23, 2010 - 17:33 UTC