Check your account profiles (and document them!)

At some point, we've all spent countless hours troubleshooting problems that seemed "catastrophic" at the time, but are resolved by a simple fix. More often than not, it turns out that the issue could have been avoided completely with just a bit of preventative maintenance. Over the course of a project, things get overlooked or taken for granted, and sooner or later, something significant gets missed.

We experienced this recently with an eBusiness Suite environment. We noticed an apparent problem with the concurrent manager service - several jobs were backing up, and none of the individual managers seemed to be running. We made the decision to restart the service, but could not get it to start properly. We also noticed that the login URL was not loading correctly, either.

So, what was going on here? No changes had been made to the system recently, and a simple restart of a service should not be this difficult. After a bit of investigation, it turned out that the APPS database account was locked. Several months ago, a database upgrade had been performed. We concluded that during that process, the default account profile (including the condition to lock an account after 10 failed login attempts) had been applied to nearly all database accounts.

The APPS, APPLSYS, and APPLSYSPUB accounts deserve special attention. When there is a problem with one of these accounts, it can affect all users of your environment. Administrators should be careful with these accounts, especially if there is potential for them to lock automatically under any circumstances. Rather than setting them to lock under certain conditions, it is generally a better practice to use strong passwords and to update them on a regular schedule. Otherwise, you risk unnecessary downtime.

As is often the case, a bit of careful inspection and documentation might have helped us avoid this problem completely. When making significant changes such as a database upgrade, it is important to fully review and evaluate the scope of the changes being made. Make notes as appropriate, and make sure they are available to all necessary personnel. This is important for providing ongoing support. A little time spent on documentation goes a long way toward minimizing problems down the road. (But you already knew that, didn't you?)