When i was driving to customer site , i was thinking that i will come back to office within 1 hour and enjoy lunch at office.But when i reached at customer site , i realized that problem seems huge as there are almost 4 VIO servers and two dedicated AIX boxes which were having almost 30 minutes hang at 581 LED during boot.There was some POWER related issue at their data center and therfore their all network equipments as well as servers faced POWER failure.
AIX servers were completing boot process , but never giving a login prompt.
VIO servers were also having successful boot , but giving some abnormal behavior.
We decided to not to start AIX client Lpars ( using VIO servers ) until we know the main reason for 581.
While searching on internet , i could not find any details about LED except that this LED code has something to do with network configuration and DNS settings.
While inuring with customer , we realized that they do have some issues with their core switch, DHCP services, DNS services and their PDC. Their windows based PDC was also crashed and was facing some problem related to network adapter driver corruption.
While wasting almost 3-4 hours at customer site , and multiple restart , i was quite sure that we have no other option except wait till their all network related issues resolve.
After 4 hours they were successful in bringing their PDC ( DHCP/DNS) up and as soon as DNS server started pinging, my all AIX Lpars, servers and VIO servers stopped hanging at 581 ( without doing any thing on AIX).
Lesson learnt:If you AIX servers are DNS clients then you are dependent on DNS server. If DNS server is down and you try to reboot AIX or VIOS then you may face issue while rebooting AIX ( LED 581).Better to put two DNS servers entry in AIX or at least put hosts=local,bind4 entry in /etc/netsvc.conf file.
When AIX OS start, LED shows 0581 and AIX just hang there:
It is because IP configaration error.
Solution:
1 Start AIX OS and choose 1st boot device as CD
2 Use AIX install CD1 to boot the OS and choose services mode
3 Choose maintain model not default install mode
4 choose default rootvg
5 choose shell with rootvg monted
6 cd /etc, and mv rc.net rc.net.orig
7 Restart OS in normal mode
8 Delete original IP config
9 Config IP with correct setting
10 mv rc.net.orig rc.net
11 Restart AIX OS in normal mode, should be OK.
The rc.net has links to ip config file(/etc/hosts ......), if rename it, the OS would`t load it when OS start.
0581 is caused by wrong ip configration.
* Source Article from : Internet
