Environment

Oracle Fusion Middleware 11g for Fusion Applications

Issue Summary

1. Set the environment, start the db listener, start the database and then use the following command to start OID's emctl agent:

cd /app/oracle/config/instances/oid1/EMAGENT/EMAGENT/bin

./emctl start agent

2. Got the following error under the logs 'emagent.nohup':

----- Fri Apr  4 18:08:32 2014::Mismatch detected between timezone in env (Pacific/Auckland) and in /app/oracle/config/instances/oid1/EMAGENT/EMAGENT/sysman/config/emd.properties (NZ). Forcing value to latter.. -----

----- Fri Apr  4 18:08:33 2014::tzOffset for NZ is 720(min), but agent is runnning with tzOffset 780(min)

3. I then tried to manually fix the timezone using various different methods but unsuccessful with all of them. EMCTL agent simply still wouldn't start and give the same timezone issue again and again.

3.1. Method 1 - see the existing timezone, update the environment variable and then update timezone. Again unsuccessful.

export TZ=Pacific/Auckland

$emctl config agent getTZ

Pacific/Auckland

$emctl config agent updateTZ

3.2. Method 2 - manually update the timezone under emd.properties file, and then try restarting emctl agent. Again unsuccessful.

export TZ=Pacific/Auckland

3.3. Method 3 - set the timezone environment variable and then run the reset tumezone command. In this case I got another error (as highlighted below).

export TZ=Pacific/Auckland

./emctl resetTZ agent

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.  

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

Updating /app/oracle/config/instances/oid1/EMAGENT/EMAGENT/sysman/config/emd.properties...

----- Sat Apr  5 00:06:13 2014::tzOffset for NZ is 720(min), but agent is runnning with tzOffset 780(min)

 -----

----- Sat Apr  5 00:06:13 2014::trying again after waiting for 1 sec to account for daylight transition

 -----

----- Sat Apr  5 00:06:13 2014::tzOffset for NZ is 720(min), but agent is runnning with tzOffset 780(min)

 -----

resetTZ failed.

The agentTZRegion in:

/app/oracle/config/instances/oid1/EMAGENT/EMAGENT/sysman/config/emd.properties

is not in agreement with what the agent thinks it should be.

Fix your environment.

Pick a TZ value that corresponds to time zone settings listed in:

/app/oracle/products/dir/oid/sysman/admin/supportedtzs.lst

Investigation Done

The issue was actually because of incorrect timezone daylight savings updates in the installed JDK and JRE.

Solution

Step 1. Simply update the timezone in your installed JDK and JRE. On the safe side I had updated the timezone for ALL the installed JDK and JRE in my entire Fusion Middleware stack. I have detailed the exact procedure in this blog here - How to use Timezone Updater Tool for updating daylight savings time for installed Java JRE and JDK? .

Step 2. Rerun the reset timezone command, which will now complete successfully.

$ export TZ=Pacific/Auckland

$ ./emctl resetTZ agent

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.  

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

Updating /app/oracle/config/instances/oid1/EMAGENT/EMAGENT/sysman/config/emd.properties...

Successfully updated /app/oracle/config/instances/oid1/EMAGENT/EMAGENT/sysman/config/emd.properties.

Login as the em repository user and run the  script:

exec mgmt_target.set_agent_tzrgn('idm117.aclnz.com:5162','Pacific/Auckland')

and commit the changes

This can be done for example by logging into sqlplus and doing

SQL> exec mgmt_target.set_agent_tzrgn('idm117.aclnz.com:5162','Pacific/Auckland')

SQL> commit

Step 3. As a part of the above reset command, it will ask you to run a script at the SQL prompt. But when you run that scipt, you will receive the following error:

SQL> exec mgmt_target.set_agent_tzrgn('idm117.aclnz.com:5162','Pacific/Auckland');

BEGIN mgmt_target.set_agent_tzrgn('idm117.aclnz.com:5162','Pacific/Auckland'); END;

*

ERROR at line 1:

ORA-20233: Invalid agent name idm117.aclnz.com:5162

ORA-06512: at "SYSMAN.MGMT_TARGET", line 3811

ORA-06512: at line 1

Step 4. According to MOS DOC ID 388280.1, the error: ORA-20233: Invalid agent name agentmachine.domain:3872 will occur if the Agent has not successfully started even once and communicated its details to the OMS. You can confirm this by querying the MGMT_TARGETS table in the sysman schema:

SQL> select target_name, target_type from mgmt_targets where target_name like '%agentmachine.domain%';

To execute the mgmt_target.set_agent_tzrgn procedure, there should be a target of type 'oracle_emd' in this table.

And in our case we could never be able to start OID EMCTL agent before, so we followed Oracle solution for this issue. The solution was to simply ignore the error and start the emctl agent (that is "Skip the steps suggested by the mgmt_target.set_agent_tzrgn routine. These are needed only if the Agent was already communicating with the OMS and its TZ has changed.").

Step 5. Now simply start the OID emctl agent with the below command and we could start it successfully.

$ ./emctl start agent

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.  

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

Starting agent ............................. started.