Would you like the schedule (based on time) Power On/Off and/or Scaling operations for your OCI resources? Well that is very easily possible with my Auto Scaling Script!
The Script supports:
- Compute VMs: On/Off
- Instance Pools: On/Off and Scaling (# of instances)
- Database VMs: On/Off
- Database Baremetal Servers: Scaling (# of CPUs)
- Autonomous Databases: On/Off and Scaling (# of CPUs)
- Oracle Digital Assistant: On/Off
- Oracle Analytics Cloud: On/Off and Scaling (between 2-8 oCPU and 10-12 oCPU)
- Oracle Integration Service: On/Off
If you want to implement this in your OCI environment, I have made a very simple instruction video:
You can also find full instructions on the github repository:
Hi,
Have created dynamic group and policy as per the video but still I am facing issue while creating name space:-
[opc@tushar-autostart OCI-AutoScale]$ python3 CreateNameSpaces.py
Logged in as: Instance Principle/comvivamfs3 @ ap-mumbai-1
Creating Namespace Schedule
Traceback (most recent call last):
File “CreateNameSpaces.py”, line 58, in
response = identity.create_tag_namespace(create_tag_namespace_details=details).data
File “/usr/local/lib/python3.6/site-packages/oci/identity/identity_client.py”, line 2388, in create_tag_namespace
response_type=”TagNamespace”)
File “/usr/local/lib/python3.6/site-packages/oci/base_client.py”, line 256, in call_api
return self.request(request)
File “/usr/local/lib/python3.6/site-packages/oci/base_client.py”, line 377, in request
self.raise_service_error(request, response)
File “/usr/local/lib/python3.6/site-packages/oci/base_client.py”, line 547, in raise_service_error
original_request=request)
oci.exceptions.ServiceError: {‘opc-request-id’: ‘8EA34C5766EF405A9E260C392069C631/B08B839C3B22E31CDAA023F3533C81C0/42F259C26D6D9C93E51A5D4F31E8F967’, ‘code’: ‘NotAllowed’, ‘message’: ‘Please go to your home region fra to execute CREATE, UPDATE and DELETE operations.’, ‘status’: 403}
Hi, is ap-mumbai your home region? The CreateNameSpaces calls can only be made to the home region. I will try to fix this week to automatically select the home region, no matter where you run the script from, but for now, run the script in an instance in your home region, or manually create the tag namespace and keys.
Hi Sir,
No, Frankfurt is the home region. I have configured on Frankfurt. Its configured on Home region successfully. Thanks.
But the thing is, scale up/down is not working other than the home region.
How I can manage the scale up/down other than the home region?
The script should find all resources across all enabled regions. Just keep in mind from a hour-of-the-day perspective, it is looking at the time of day of where the script is running, not where the resource is located.
If you have created a resource in an other region with the schedule tag, do you see it in the output of the script come by?
Yes, I have configured the python script in Frankfurt and Tag the resource of Mumbai & Frankfurt both region. But Its working for Frankfurt only.
Even when I am running it manually using “paython3 AutoScaleALL.py”. Its not considering the Mumbai related resources in the output of script and resource are not scaling down.
Hello Richard,
[opc@compins01 OCI-AutoScale]$ python3 CreateNameSpaces.py
Traceback (most recent call last):
File “CreateNameSpaces.py”, line 11, in
import oci
ModuleNotFoundError: No module named ‘oci’
Did I forget a step? Do I need to make a copy of config_example with my own OCIDs?
did you run the “pip3 install oci” command?
You are missing the OCI library
Got it working on a fresh new Compute Instance. I re-used an old one – my mistake.
Super! Enjoy the script. Hope it can help to make savings 🙂
Richard, thank you for this great gift to the community; I plan to make very good use of it and save lots of $$$.
I’ve been using it successfully for “AnyDay/Weekday” tags, but I’m having trouble getting the specific day of the week tag (i.e. Monday, Tuesday, etc.) to work. Is it a known issue, or am I mis-configured somehow ?
When used alone, I get “not correct amount of hours”
When used with more general WeekDay/AnyDay tags, it seems to not be overruling as expected.
Schedule
Tuesday
1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,1,0,0
Checking: ccca – Instance
Error with schedule of ccca – , not correct amount of hours
WeekDay
1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,1,0,0
Tuesday
1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1
Checking: ccca – Instance (on a Tuesday)
– Active schedule for ccca : 1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,[0],0,0,1,0,0
“A specific day of the week tag (ie. Monday) overrules all other tags.”
Hi, if I look at both WeekDay and Tuesday schedule, they have the same number for the current hour. The last 6th number are both a 0.
Maybe you clock is running in a different timezone then you expect it to be?
You can see, based on the [ ] which element is being used.
ok, I will double check on the timezone for the double schedule, but why do I get
“not correct amount of hours” with any of the named weekdays when used by themselves ?
I’m pretty sure I have 24 numbers (have recounted, and have cut/paste from a working AnyDay string.
This is the schedule I’m trying now to shutdown at 5pm Friday and startup first thing on Monday
(does it look reasonable?)
Schedule.WeekDay: 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
Schedule.Friday: 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0
Schedule.Weekend: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
Looks perfectly fine. will investigate.
I think the problem has been fixed, the specific day check was one tab to much indented. Please refresh the code and all should be good. If not, let me know 🙂
I count 24 as well 🙂 I will check. I am working on the script any way this weekend, as I am also adding support for load balancer scaling and exadata. Finally convinced oracle to provide me with exadata 🙂
HI Richard, thanks for your VDO and github link. I find this is useful. However, i’ve been observing some ‘off’ time delayed. For instance,
WeekDay : 0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0
-> I want it to on at 10am, and off at 3pm.
but it is on at 10am and off at 5pm.
Anything I need to correct it?
Appreciate your help on this.
Any thoughts on this one?
Thank You
[opc@autoscalertaskscheduler OCI-AutoScale]$ sudo python3 CreateNameSpaces.py
Logged in as: Instance Principle/philipsradoncoci @ phx
Creating Namespace Schedule
Traceback (most recent call last):
File “CreateNameSpaces.py”, line 58, in
response = identity.create_tag_namespace(create_tag_namespace_details=details).data
File “/usr/lib/python3.6/site-packages/oci/identity/identity_client.py”, line 2388, in create_tag_namespace
response_type=”TagNamespace”)
File “/usr/lib/python3.6/site-packages/oci/base_client.py”, line 260, in call_api
return self.request(request)
File “/usr/lib/python3.6/site-packages/oci/base_client.py”, line 375, in request
self.raise_service_error(request, response)
File “/usr/lib/python3.6/site-packages/oci/base_client.py”, line 545, in raise_service_error
original_request=request)
oci.exceptions.ServiceError: {‘opc-request-id’: ’50BD1EB02CFE4BF9B878DA0D6B06CEE6/0E1EFB9D015C9E88202A3471C2DE5A8F/18BFEB2110145331558DE23FAC438E05′, ‘code’: ‘NotAuthorizedOrNotFound’, ‘message’: ‘Authorization failed or requested resource not found’, ‘status’: 404}
[opc@autoscalertaskscheduler OCI-AutoScale]$
Does this dynamic group has the permissions to create a namespace definition?
I have the same error. I am using VM . I have this for the Policy
allow dynamic-group wl-autoscaling to manage all-resources in compartment prod.
Hi Richard,
I have this for the Policy
allow dynamic-group SchedulePermissions to manage all-resources in tenancy
Hi Richard,
It seems like this doesn’t work for Bare Metal machines.
I was able to get it to work with Virtual Machines.
Thoughts?
Thank you
Correct. You always have to pay for BM servers, no matter if your turn them on or off. So that is why the auto scale script does not support them.
Richard,
This is great tool! We have been using it on our Windows compute instances, and our VM ASM DB Systems. We recently started using it on LVM (vs ASM) based DB Systems, and the database start up is not working properly. It is not turning on all the listener services, in particular the service for the pdb. I’m curious what commands are used to turn on and off the db and db systems.
sounds more like a bug 🙁 Did you file an service request for this?
I am using the OCI Auto-scale to stop/start an Autonomous Data warehouse with the following schedule…
Weekdays: 0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0 (Power On 9 am to 6 pm)
For some reason, the schedule is not stopping the database at 6pm, I enabled the notifications options and getting this message for the Scale Down procedure:
Scaling (Down) just completed. Found 0 errors across 0 scaleable instances (from a total of 1 instances).
Error Details: []
Success Details: []
If I run AutoscaleALL.py then it shutdowns the database, I received the following message:
Scaling (All) just completed. Found 0 errors across 1 scaleable instances (from a total of 1 instances).
Error Details: []
Success Details: [‘ – Initiate Autonomous DB Shutdown for ADB’]
Any thoughts
Thanks!
Hi Rishard,
I set up the script to start/stop an Autonomous Data warehouse instance with the following schedule:
WeekDay: 0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0 <– Available from 9 AM – 6 PM
Weekend: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 <– Unavailable during weekends
The Linux VM is set up to America/Toronto timezone
The script can successfully start the ADB at 9 AM but it fails to stop it at 6 PM. I configured the script to send notifications and it seems it can find the schedule or instance based on the email I received.
Scaling (Down) just completed. Found 0 errors across 0 scaleable instances (from a total of 1 instances).
Error Details: []
Success Details: []
If I run AutoScaleALL then ADB is shut down successfully, the notification I receive is the following:
Scaling (All) just completed. Found 0 errors across 1 scaleable instances (from a total of 1 instances).
Error Details: []
Success Details: [' – Initiate Autonomous DB Shutdown for SampleADB']
Any ideas?
Best regards,
Jorge
Hi Richard,
Great tool and many thanks for sharing this.
I first of all ran a successful deployment on my personal Oracle Cloud Free Tier account. The script was able to control compute instances and an ATP database instance perfectly without issue.
I have since deployed to my employer’s PAAS account which has the home region set to Germany Central (Frankfurt). I created the AutoScaler compute instance in the home region and the script can control another compute instance and 2 ATP databases in the same region. All good.
However, most of our database and compute instances are in the UK South (London) region and the script is not able to see these tagged instances at all. All the instances in the UK have been tagged with the same values that work for the instances in Frankfurt and the script fails to reference them at all.
Script output –
[opc@autoscaler OCI-AutoScale]$ python3 /home/opc/OCI-AutoScale/AutoScaleALL.py up
Using Instance principle
Starting Auto Scaling script, executing Up actions
Logged in as: Instance Principle/******* @ eu-frankfurt-1
Day of week: Monday – Weekday: True – Current hour: 14
Checking INTEGRATION_TEST – AutonomousDatabase…
Checking: INTEGRATION_TEST – AutonomousDatabase
– Active schedule for INTEGRATION_TEST : 1,1,1,1,1,1,1,1,1,1,1,1,1,[1],1,1,1,1,1,1,1,1,1,1,
– Starting Autonomous DB INTEGRATION_TEST
Checking INTEGRATION_DEV – AutonomousDatabase…
Checking: INTEGRATION_DEV – AutonomousDatabase
– Active schedule for INTEGRATION_DEV : 1,1,1,1,1,1,1,1,1,1,1,1,1,[1],1,1,1,1,1,1,1,1,1,1,
– Starting Autonomous DB INTEGRATION_DEV
Checking instance-********-**** – Instance…
Checking: instance-********-**** – Instance
– Active schedule for instance-********-**** : 0,0,0,0,0,0,0,0,0,0,0,0,0,[0],0,0,0,0,0,0,0,0,0,0,
All scaling tasks done
So the script can see the tagged instances located in the home region but not the tagged instances located in UK South (London) and presumably any other region.
Any ideas?
Many thanks.
Yes, currently the script only works for a single region. I am working on making it support multiple regions, I am just trying to figure out the best way to deal with time zones.
Hey Richard ,
I fixed the previous namespace error, my namespaces are created . But, after running the command #python3 AutoScaleAll command I got the following error,
Using Instance principle
Starting Auto Scaling script, executing All actions
Traceback (most recent call last):
File “AutoScaleALL.py”, line 258, in
goldengate = oci.golden_gate.GoldenGateClient(config={}, signer=signer)
AttributeError: module ‘oci’ has no attribute ‘golden_gate’
Plz help me to fix this.
Regards,
Harish
You have an old version of the OCI python SDK. Please run the command:
pip3 install –upgrade oci
Hi Richard,
I am getting below error message. when run AutoScaleALL.py.
[opc@autoscaling-vm-thilina OCI-AutoScale]$ python3 AutoScaleALL.py
##########################################################################################
# Running Auto Scale #
##########################################################################################
Traceback (most recent call last):
File “AutoScaleALL.py”, line 1490, in
config, signer = create_signer(cmd.config_profile, cmd.is_instance_principals, cmd.is_delegation_token)
File “AutoScaleALL.py”, line 222, in create_signer
tenancy=config[“tenancy”],
KeyError: ‘tenancy’
[opc@autoscaling-vm-thilina OCI-AutoScale]$
Hi,
Did you configure the OCI CLI with the correct authentication in the config file or are you trying to use instance principle as authentication?
How does this work with E3.Flex instances, where you specify not just the CPU but also the memory ? I could just start and stop an instance, but if you upscale E3, how does it handle the memory change (unlike E2 which have set shapes)
good question 🙂 currently the script does not support this, but I think we can implement something.
so you want to be able to specify cpu and memory for each hour, and based on that reshape the instance.
you do known that this reshaping does require a reboot
Yeah, I understand it requires a reboot. I guess with IaaS you can scale vertically or horizontally. Horiszontally allows you to add more nodes without an outage, but vertically would allow you to scale an IaaS instance (not sure how useful this would be… and TBH, up/down is good enough for me just now).
Actually, it looks like the scaling doesn’t work with IaaS instances at all, just up/down.. Is that something you’re planning to add ?
I had a suggestion. We have a team who wants the system shut down every Friday evening (if it’s running), but then left down (and they will restart it only if they need). The problem is that if you use 1,1,1….0,0,0,0 then it will start if it’s down, which isn’t what we want.
The suggestion is, could you add “-” as an option (rather than an integer), which means “no change”, this would mean
-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,0,0,0,0,0,0,0,0
Would do nothing for the first few hours of the day, but then then shut down in the evening.
Great suggestion and this has already been implemented. You can use a * for this 🙂
Hadn’t spotted that.. I’ll give it a try 😉
Hey Richard
I am trying to implement this solution for a small subset of my instances. I am getting the error below
Traceback (most recent call last):
File “AutoScaleALL.py”, line 1490, in
config, signer = create_signer(cmd.config_profile, cmd.is_instance_principals, cmd.is_delegation_token)
File “AutoScaleALL.py”, line 219, in create_signer
(config_profile if config_profile else oci.config.DEFAULT_PROFILE)
File “/usr/lib/python3.6/site-packages/oci/config.py”, line 106, in from_file
raise ConfigFileNotFound(“Could not find config file at {}, please follow the instructions in the link to setup the config file https://docs.cloud.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm“.format(expanded_file_location))
oci.exceptions.ConfigFileNotFound: Could not find config file at /root/.oci/config, please follow the instructions in the link to setup the config file https://docs.cloud.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm
Following the github and youtube steps I cannot see anything for the .oci config file although I can see an example of this on the github repo. Any info would help a bunch.
sounds like your did not configure the authentication of the sdk
run the command: oci config setup
or see instructions on github page
(posted on git as well)
Found a weird problem..
If I have a tag: *,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*
Then this works if I select AnyDay, but If I select Friday it says I’ve not entered enough values:
Checking instance-20210817-0927 (Instance)…
– Active schedule for instance-20210817-0927: *,*,*,*,*,*,*,[*],*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,*,
– Ignoring this service for this hour
Checking cgretailsapps (Instance)…
– Error with schedule of cgretailsapps – , not correct amount of hours, i count 1
(these 2 instances have the same tag, one works – AnyDay, one doesn’t – Friday)
I found this bug and fixed it in the script. Please update to the latest script available on github.
Hi Richard,
Any idea on this error. Please suggest if I missed any step –
##########################################################################################
# Running Auto Scale #
##########################################################################################
Starts at 2021-12-28 11:10:40
Connecting to Identity Service…
Traceback (most recent call last):
File “AutoScaleALL.py”, line 1557, in
tenancy = identity.get_tenancy(config[“tenancy”]).data
File “/usr/lib/python3.6/site-packages/oci/identity/identity_client.py”, line 6569, in get_tenancy
response_type=”Tenancy”)
File “/usr/lib/python3.6/site-packages/oci/base_client.py”, line 457, in call_api
response = self.request(request)
File “/usr/lib/python3.6/site-packages/circuitbreaker.py”, line 52, in wrapper
return self.call(function, *args, **kwargs)
File “/usr/lib/python3.6/site-packages/circuitbreaker.py”, line 67, in call
result = func(*args, **kwargs)
File “/usr/lib/python3.6/site-packages/oci/base_client.py”, line 586, in request
self.raise_service_error(request, response, service_code, message)
File “/usr/lib/python3.6/site-packages/oci/base_client.py”, line 748, in raise_service_error
original_request=request)
oci.exceptions.ServiceError: {‘opc-request-id’: ‘FC6AB8320CFC4CE09B2849D3B82A9BBD/646C39A7DD75347D6B1541AF6558AF6B/9B731078C12AB70FF173CB999DCFE53A’, ‘code’: ‘NotAuthenticated’, ‘message’: ‘The required information to complete authentication was not provided or was incorrect.’, ‘status’: 401}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “AutoScaleALL.py”, line 1582, in
raise RuntimeError(“\nError connecting to Identity Service – ” + str(e))
RuntimeError:
Error connecting to Identity Service – {‘opc-request-id’: ‘FC6AB8320CFC4CE09B2849D3B82A9BBD/646C39A7DD75347D6B1541AF6558AF6B/9B731078C12AB70FF173CB999DCFE53A’, ‘code’: ‘NotAuthenticated’, ‘message’: ‘The required information to complete authentication was not provided or was incorrect.’, ‘status’: 401}
[opc@autoscaler-instance-20211224-1603 OCI-AutoScale]$ client_loop: send disconnect: Connection reset
I found the solution to the above error.
I have to use sudo while running python3 AutoScaleALL.py command.
————-
[opc@autoscaler-instance OCI-AutoScale]$ sudo python3 AutoScaleALL.py
##########################################################################################
# Running Auto Scale #
##########################################################################################
Starts at 2022-01-11 12:15:26
Connecting to Identity Service…
Version : 2021.08.17
Command Line :
Tenant Name : ….
Tenant Id : ocid1.tenancy.oc1…
Home Region : us-ashburn-1
Action : All
Tag : Schedule
Loading Compartments…
Total 33 compartments loaded.
##########################################################################################
# Region us-ashburn-1 #
##########################################################################################
Starting Auto Scaling script on region us-ashburn-1, executing All actions
Day of week: Tuesday, IsWeekday: True, Current hour: 7
Getting all resources supported by the search function…
Finding MySQL instances in 33 Compartments…
………………………….
Checking 1 Resources for Auto Scale…
Checking Event_Functions_instance-20211223-1621 (Instance)…
– Active schedule for Event_Functions_instance: 0,0,0,1,1,1,[1],1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,
Waiting for all threads to complete…
Region us-ashburn-1 Completed.
##########################################################################################
# Region us-phoenix-1 #
##########################################################################################
Starting Auto Scaling script on region us-phoenix-1, executing All actions
Day of week: Tuesday, IsWeekday: True, Current hour: 4
Getting all resources supported by the search function…
Finding MySQL instances in 33 Compartments…
………………………….
Checking 0 Resources for Auto Scale…
Waiting for all threads to complete…
Region us-phoenix-1 Completed.
All scaling tasks done, checked 1 resources.
——————————
If you have to do sudo, it probably means the python OCI modules where not installed for this user. You should not have to sudo to run this script.
The message says notAuthenticated, so I assume your OCI API authentication is not correctly setup.
Are you using the config file or instance principle? Can you run the commmand: oci os ns get
I am using config file. Any guide link for setting up OCI API authentication?
I did setup the OCI CLI as explained in the below link – >
https://avmconsulting.net/oci-command-line-setup-and-some-interesting-features/
Please see the output of <>command –
[opc@autoscaler-instance]$ oci os ns get
ServiceError:
{
“code”: “NotAuthenticated”,
“message”: “The required information to complete authentication was not provided.”,
“opc-request-id”: “iad-1:T4nnECyOJxXf_QPa5ONPyvSl6g8EPPF7MHa7Xn5KKZHtMj3IS8PxKxF5UlgmDOT5”,
“status”: 401
}
sounds like you have not propely configured the autentication. Did you add the API key to the user account and does the user have enough permissions?
I created a fresh instance , configured the CLI and added the API Key to my user. Now I am able to run the script without ‘sudo’.
Thanks for your help.
Hi Richard,
Could you please show how to schedule the script to run every hour?
Thanks
Mausami
Hi Richard,
Great solution. I have deployed this on a group of our test servers but have noticed some strange behaviour
We are using EDT timezone with the following tags
Schedule.Friday: 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0
Schedule.Saturday: 0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
However, the instances are rebooting at midnight Thursday/Friday
I have tested and change the tag to below but the error persists.
Schedule.Friday: *,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0
Any possible ideas of what might be causing this?
when the script runs, at which number do you see the brackets? [ ]
Hi Richard, thanks for taking the time to get back to me on this. Unfortunately, due to my timezone UTC and the timezone EST the script is running in I am unable to observe this action.
As a test i have spun up an instance and ran the script with the output below
Current time est is 08:50
##########################################################################################
# Region us-ashburn-1 #
##########################################################################################
Starting Auto Scaling script on region us-ashburn-1, executing All actions
Day of week: Thursday, IsWeekday: True, Current hour: 9
Getting all resources supported by the search function…
Finding MySQL instances in 11 Compartments…
………..
Checking 4 Resources for Auto Scale…
Checking test isntance auto scheduling (Instance)…
– Active schedule for test isntance auto scheduling: *,0,0,1,1,1,1,1,[1],1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
Checking XXXXXXX (Instance)…
– Ignoring instance, as no active schedule for today found
Checking XXXXXXX (Instance)…
– Ignoring instance, as no active schedule for today found
Checking XXXXXXX (DbSystem)…
– Ignoring instance, as no active schedule for today found
Waiting for all threads to complete…
Region us-ashburn-1 Completed.
Yes see in the script is says: current hour 9, but it has [ ] around 8th hour. Will look in to code to see what the issue is.
Can you test this again with the latest version of the script please