Troubleshooting Environment Problems Using Admin Data Tools
Overview
This blog is written for Teamcenter administrators who are interested in finding the latest and greatest tools for troubleshooting environment issues. Tracking down an environment issue can sometimes feel like Alice falling down into the rabbit hole. Often times you may consider redeploying the latest patch or custom software to avoid having to dig through all administrative configurations to search for possible causes of the issue.
Teamcenter 11.2 offers new tools called Administrative Data Tools that greatly reduce the burden of administrative tasks. Tasks become much easier with these tools such as copying all admin data from one environment and importing into another environment, analyzing the administration configurations in a single environment, or comparing the administration configurations between two environments. In this post we are going to demonstrate the high value to your organization of comparing two environments using the Admin Data Tools. This demonstration will be performed using a light hearted yet very real scenario.Let us begin.
Backstory
Let us say that over the last several months your team has been working on a rollout plan to update your production system with the latest custom software. Your team developed the custom configurations using a Teamcenter Development environment, then moved the configurations up to a Test environment, and into a Training environment. And finally this past weekend the Production environment was updated.
It is now Monday, the system is up and running and users are on the system and you are the on-duty lead administrator. Its 4:00 PM and you are thinking about tonight’s dinner with your wife or husband. She reminded you this morning to be home and ready by 6:00 PM so you will have to leave work soon. You start thinking about the cheesecake you are going to eat tonight when the admin desk phone suddenly rings and brings your thoughts back to work.
You answer the phone. It is a Teamcenter user calling, Mark Johnson. Mark is saying that he cannot view the IP data (Intellectual Property) that he used to be able to view in Training environment. You write this down and a few minutes later the phone rings again. This time a user is complaining that “Item Query” is not returning the correct data that it did during the testing of the rollout using the Training environment. Again you write this down and start to think about how to perform an investigation.
The phone rings five minutes later and a group of users are stating that they cannot see some of the Revision Rules choices in the Product Structure Editor application in the Production environment that there were able to see previously in the Training environment. You start forgetting about the cheesecake and you start to get that tight feeling that something is not right.
Your cell phone rings and you know that most people do not know your cell phone and so you start to feel nervous as you pick it up. It is the vice president and he is stating that many users are experiencing performance problems in the Production environment. You had just promised the VP a few days before that all performance problems had been worked out and the system would perform as expected. However, you know something is wrong and you are starting into a full panic because problems like these four issues can take hours to solve if not days.
One of the options you come up with is to re-install the custom software update that was performed over the weekend. Perhaps a step was missed during this process. You start to consider whether the custom software install should be performed again as the quickest solution. However you know that that will have to start now and could take many hours to complete, not to mention the impact of downtime. You are about to call your wife when you remember reading this Teamcenter blog about how use the Admin Data Tools for troubleshooting environments.
You quickly put your cell phone back down and tell your team to export the Admin Data from both the Training and Production environments using the admin data utilities (listed below). You choose these two environments because everything was working great in the Training environment and the issues are being reported about the Production environment. So your goal is to compare these two environments to analyze the differences in administration configurations.
You are not really sure what to look for so you have all admin data exported so that you can look for differences that stand out as potential candidates that are causing these issues.
So your team exports admin data from the Production environment using this command:
admin_data_export –adminDataTypes=all …
Then your team exports admin data from the Production environment using the same command in the other environment:
admin_data_export –adminDataTypes=all …
Next your team generates the “Admin Data Compare Report” by running the command below. The command below will load the two exported admin data packages that were exported from the Training and Production environments and then generates the compare report.
generate_admin_data_compare_report –adminDataTypes=all …
Note see the help text that comes with these commands for more information on how to configure the command arguments.
The Analysis
Open the generated Admin Data Compare Report and start analyzing the differences. The first page of the Compare Report is the summary page which gives you a big picture view of the differences. Before starting into the analysis, you first verify that this report is showing the differences between the two environments you expected to see. This is verified by looking at the top of the report where the label states “Training vs Production” (See the top of the picture below). You could also click on the Report Details link (far left side) to get more details about the exported data: timestamp of when the export occurred, Site IDs, and Site Name values for each database.
The next aspect to notice is the tiles and color highlighting. Each tile represents one Admin Data category. Categories are available for Access Manager, Organization, Preferences, Revision Rules, Saved Queries, Projects, Subscriptions, and Workflow. Note that some of the tiles are highlighted in a yellow background and others have a white background. The tiles that are highlighted in yellow have differences in their areas. The tiles with a white background have no differences. Therefore from the summary page you can already decipher that there are differences in the categories of Access Manager, Organization, Preferences, Revision Rules, and Saved Queries.
Remember that we are looking for potential candidates and we are using this report to see what differences could be causing the issues. Therefore we are off to a good start as we have identified some areas that could contain potential candidates by focusing in the categories that have a yellow background.
Access Manager
First we start our investigation in the Access Manager category. Note that when you look at a tile in the Summary Page, the interior of the tile includes the names of the various Admin Data elements that belong to the category. Next to each element name is a number under the “Differences” column. This information is displaying the number differences between the two environments for each element. In the case of the Access Manager category, there are 0 Rules with differences, which means the Rules in the Rule Tree are identical in both environments. This means the differences in behavior between the two environments cannot be caused by Access Manager Rule differences because there are no differences. The Named ACLs element shows 1 difference, and the Privileges element displays 0 differences. Therefore the Summary Page is very useful in getting a high level view about the types of differences between the two environments. This information helps us to determine where to dig deeper for more information.
Clicking on the Access Manager tile (or the Access Manager menu in the left side menu strip) will take you to the comparison of Access Manager (see picture below).
Also notice that in this view of the Access Manager, the bottom left area of the Compare Report displays the menu list of elements within the Access Manager category. As you can see, the element names, differences count, and yellow/white color highlighting are also displayed here. And as you can determine the yellow highlighting is displaying the elements that have differences.
Since the “Named ACLs” has the yellow highlighting we click on the “Named ACLs 1” menu link to see details about these differences. Looking at the report we can easily find the one rule called “System” that is different between the two environments as it is highlighted in a yellow background color. Clicking on the “System” Named ACL the right side of the report displays the various properties of the System Named ACL (see picture below). If you click on the “Differences” menu near the top of the page on the right side panel, the report will hide all of the properties that are equal and display only the properties that are different. In this case we can narrow in and see that the “CHANGE_OWNER” privilege has a value of “Revoke” in the Training environment and a different value of “Grant” in the Production environment. This is peculiar that this privilege could have been flipped from Revoke to Grant by someone, but it does not explain any of the four problems that we received complaints about. Perhaps another user will call about an access issue related to Change Owner later. However right now this is not related to one of the four issues, so we will make a note and look into this later.
Remembering that we are hunting for potential causes of the issues, we will move out of the Access Manager category because it does not look like this difference is causing the issues we know about.
Organization
Next we click the “Summary” menu to go back to the Summary page, and then click the Organization tile to drill down in the same way that we just did with the Access Manager category. The Summary Page showed us that the Organization category has differences in the “Users” element so we start there by clicking on the “Users” element in the sub menu in the bottom left hand corner. We can see in the report that two users have differences “DasAtish” and “Mark Johnson”. We click on DasAtish first and see that his Status was changed from “Inactive” to “Active” (see picture below). This does not sound like a likely cause for any known problem so we ignore it.
Then we click on the other user “Mark Johnson” and look to the right to see the list of user properties that have changed for this element (see picture below). Seeing Mark Johnson’s name is interesting because Mark was the first user to call in with a problem today and he stated that he was not being able to view Intellectual Property data. On the right, we can see that Mark’s Geography property was changed from “US” to “CA”. This is likely the cause because there are custom rules added by your company to the Access Manager that control Read access based on the User’s Geography settings. So it looks like we have found the first issue. You write it down and keep moving to the next category.
Preferences
Moving on to the next category that had differences, we click on the Preferences tile on the Summary Page. While the Preferences sub-menu’s (shown in the picture below in the lower left corner) yellow highlighting shows that there are differences in the area of the Preferences, Preference Values, and Site Overrides, we click on the Preference Values menu first. The Preference Values page displays only the Preference values that are different. In contrast the other menus in this area show differences that can include preference values but also can include differences such as preference descriptions. Since a change in a preference description is not likely to cause any of the issues our callers indicated, we go first to the Preference Values page to see if anything catches our eye.
The report shows that there in only as single preference whose value is different. The “TC_Journaling” preferences is set to “OFF” in Training environment and is set to “ON” in the Production environment. Looking to the right side panel we read the preference description and find that this preference controls additional information being included in the log files. The Vice President did say that a number of users were complaining about performance and it is known that if the system is trying to log more information into a log file while users are interacting with the system, it can cause the system to be sluggish. While you do not really know why this preference was changed from OFF to ON, we can assume that an Admin went in and turned it on. Possibly to check out an issue and then forgot to turn it back off and therefore this explains the sluggish performance complaint. So we take a note of this preference and move on to the next category.
Revision Rules
Moving on to the Revision Rules tile we follow a similar procedure of analysis. Our first look at Revision Rules shows that there are a fair number of revision rules that have differences. Clicking on any of the yellow highlighted Revision Rules (see picture below), then looking to the right side of the report at the Comparison Details area we see that the Owner was set to “infodba” in the Training environment and was changed to “tcadmin” in the Production environment.
We know that when the custom Revision Rules are loaded into the system via the scripts, they are loaded using the “infodba” account. As a part of your custom software install steps, there is a subsequent step where all Revision Rules have their ownership changed to “tcadmin”. This is performed because there are rules in the Access Manager that control who has “Read” access to the Revision Rules. The Read access thus controls the users who can view the various Revision Rules as choices in the Product Structure Editor application. The Access Manager Rule bases the Read access by who the owner of the Revision Rule.
Thinking that this could be the issue, you can quickly click on the “Ownership” submenu (see the menu in the picture below at the bottom left side) to see a list of all Revision Rules, the owning user, and the owning group in a single view.
This view makes it easier to view all ownership information for troubleshooting ownership issues like this one. From this view we can quickly ascertain that the Owning User has been changed for six of the Revision Rules and this is the issue that one of the callers called in about. So we take a note and move on to the next category to find the last issue.
Saved Queries
Clicking on the Saved Queries tile brings us to a view of all Saved Queries. To see a list of only the Saved Queries that have differences, click the “Differences” link at the top of the page. The report now shows that there is only one Saved Query called “__Item_Name_Query” that has a difference between the two environments (see picture below). This is already perking our interest because one of the callers complained about the “Item Query” not working correctly.
Click on the “__Item_Name_Query” query and look to the Comparison Details displayed on the right to analyze the properties of this query that are different (see picture below). Immediately the eye is drawn to the yellow highlighting on the “Query Clauses” property.
Upon inspecting the Query Clauses field closely we can see that the Training environment has set the following ‘WHERE “item_id”…’, whereas the Production environment has it set to ‘WHERE “item_iddd”…’. We can see that someone has edited the query clause, perhaps accidentally, to an attribute name that is not valid.
Issue Resolved
Confident that you have identified the problem you quickly take a note and send emails to you administration team to quick fix the four issues you found in the admin data comparison report. Then you send a quick email to the Vice President letting him know that the performance problem has been found and was simply a preference set incorrectly and that the problem is being corrected immediately.
Rather than taking the system down and taking hours to reinstall the custom software, instead you used the Admin Data Compare report. All problems were solved in 10 minutes. You relax and finish your coffee. Then you head home in time to go out with your wife and each your favorite dessert.
About the Author
Bob Donovan works in Teamcenter Product Management for Siemens PLM Software with 18 years of experience in the PLM industry. Bob’s areas of responsibility are in the foundation area of Teamcenter and include: Software Deployments, Administration Data, Data Model and coding extensions to Teamcenter, SOA, BMIDE, LOVS, Security, Authorized Data Access, ITAR, Forward and Reverse Proxies, Single Sign On, Access Manager, Organization, Logging and Monitoring, Subscriptions, Projects, Smart Folder, Install, and Upgrade.
