Cloud | Cloud Security | Security
Favorite Google Cloud Service: Google Cloud Security Command Center Premium Share 
Startup scripts are used to configure or initialize a Virtual Machine at boot or provisioning time. On hardened machines – with read only file systems – your scripts will fail to run. This blog show you why, and how to go about this.
Detect failed startup scripts
Startup scripts fail silently.. Therefore, you first notice that the VM health check fails, just to detect that the startup script simply failed to provision or execute.
In the next example a startup script fails to execute, due to invalid file system permissions. The /tmp-file system is mounted with noexec – as recommended per CCEID 1.1.9 or CIS CSCv7 14.6 – hence executables can’t run. Interestingly, however, the google-startup-script.service is said to succeed.
...
Jun 14 10:24:31 laurens-vm google_metadata_script_runner[1690]: startup-script: /bin/bash: /tmp/metadata-scripts052481810/startup-script: Permission denied
Jun 14 10:24:31 laurens-vm google_metadata_script_runner[1690]: startup-script exit status 126
Jun 14 10:24:31 laurens-vm google_metadata_script_runner[1690]: Finished running startup scripts.
Jun 14 10:24:31 laurens-vm systemd[1]: google-startup-scripts.service: Succeeded.
...
Startup script execution flow
The Guest Agent for Google Compute Engine runs your startup scripts. More specifically, the google-metadata-script-runner-component gets the startup-script-metadata attribute, saves the inline script to the run_dir-folder and executes the temporary script file.
For cloud storage backed startup scripts,
startup-script-url-metadata attribute, the process downloads the script to the temporary script file.
Startup script execution consists of two stages: script file provisioning, and script file execution. Script provisioning fails if you provision to a readonly file system. Script execution fails if you execute on a noexec file system.
It is not feasible to alter the script execution flow – although I encourage you to contribute to GitHub. Therefore, let’s find a way to configure the startup script file location. If we can configure a writable, executable location, all is good.
Configure startup script file location
Configure startup script file location using the Guest Agent configuration file. More specifically, set the run_dir-setting to a writable and executable file system in your image creation process.
The next example sets the run_dir-setting to /var/lib/google, which is assumed to be writable and executable.
mkdir -p /var/lib/google
sed -i 's|run_dir =|run_dir = /var/lib/google|g' /etc/default/instance_configs.cfg
Boot your updated image to verify that the startup script runs.
...
Jun 14 10:24:31 laurens-vm google_metadata_script_runner[1690]: startup-script: Hello World!
Jun 14 10:24:31 laurens-vm google_metadata_script_runner[1690]: startup-script exit status 0
Jun 14 10:24:31 laurens-vm google_metadata_script_runner[1690]: Finished running startup scripts.
Jun 14 10:24:31 laurens-vm systemd[1]: google-startup-scripts.service: Succeeded.
...
When you can’t change the startup file location
Your image is immutable, brilliant! Most likely you don’t need to run a startup script. However, for the sake of the article, consider installing the startup script as a service during your image creation process. Check these guides to learn how to do so on Linux using SystemD or on Windows using Scheduled Tasks.
Conclusion
Startup scripts allow for additional VM configuration. The scripts run by first provisioning the script file and later executing it. With this workflow, startup scripts silently fail to run when your file system is read only or non executable. Gladly, you can configure the Guest Agent to use an appropriate file system.
As a cloud consultant I enjoy taking software engineering practices to the cloud. Continuously improving the customers systems, tools and processes by focusing on integration and quality.
