I was actually troubleshooting some other issue for which I had to login to platform service controller (PSC) via SSH service. When i logged in to PSC VM admin console in browser (https://<PSC IP>:5480) using logon credentials, I saw a strange warning on the home screen. Overall health was showing warning and there was a heath message as below,
It was stating as “/storage/log filesystem is low on disk space or nodes”, immediately i looked at the SSO service, it was in running state, I didn’t had any issue triggered from PSC VM because of this warning. However, my earlier experiences regarding to no disk space issues which inturn got me into a strange situations pushed my mind to resolve this issue before proceeding to look into the other issue which I was actually trying to resolve.
So to analyse it, I enabled SSH and bash services from the same console, then logged into PSC VM using SSH service. Once I logged in, my 1st thought was to check the disk space utilized by different PSC file systems. So entered command df -h
As the warning was actually regards to /storage/log path, my eyes immediately looked into the /storage/log file system path. It was almost utilized 96% out of allocated 5 GB storage space.
Then i thought, let me increase the virtual HDD size from the VM edit settings, when i did that to do the changes, options to increase the provisioned space was grayed out !!. That means to increase to vHDD size, PSC VM is supposed to be in a shutdown state. Since I was not in a situation to shutdown the PSC VM as it was a primary VM and setup was in production, I thought to check some other solution to resolve.
While surfing online, I found VM KB article KB2143565 which made me to understand the actual issue and how to resolve.
It was stating that,
“This issue occurs due to an decrease in the SSO log files in the /storage/log directory. During the log rotation, the old log file is not compressed resulting in multiple large files in the /storage/log directory”
the actual workaround for this is to modify the log4j.properties file. The major modification includes to decrease the maximum backup size and maximum backup index size.
So As per the instruction, navigated to below path in PSC SSH connection,
listed the files located under the above path directory, I found the log4j.properties which i wanted to modify, when i edited the file using VI editor, i could able to find different section of codes respect the different types of logs.
Under ‘logfile’ section, I did the following changes, MaxIndex to 5 and MaxFileSize to 50000000 (50 MB)
Note: default values are mentioned in the above screenshot, which was 10 and 100000000 (100 MB) respectively.
Also, under PERFLOG section, I did the following changes, max index to 3. Default value was 5.
So modification were done as per the instruction and saved the file.
then restarted vmware-stsd service using command for changes to take effect,
service vmware-stsd restart
later, navigated to following path
log files were too many as shown below
now removed old logs to reduce the storage space. Old logs associated to localhost_access_log, vmware-identity-sts and vmware-identity-sts-perf log files should be removed.
again restarted, vmware-stsd service
service vmware-stsd restart
Uffffffff, such a long procedure !! all changes required to be done were done. I verified the storage space utilization later,
Cool, this time /storage/log size got reduced to 57%, almost 50% reduced.
All done. then I validated the PSC GUI console, i didn’t see any warning and health was showing good in green color 🙂
Actually, we don’t need to do all these steps to resolve this issue, we can also resolve it by increasing the respective virtual hard drive and expand the drive within PSC VM, which will inturn clear the warning. Some situations where we don’t additional storage space, it needs lots of approvals, or in some situations where you don’t get option to change the drive size as the option itself would be grayed out, and you will not be in a position to power down the PSC VM. In such scenarios, you can use the above instructions to resolve the issue.