Initial push to Github for Oracle Developer Day VM Builder

Author: bamcgill

OTNLicense.txt

@@ -0,0 +1,3 @@
+WARNING: THIS VIRTUAL MACHINE IMAGE CONTAINS ORACLE PROPRIETARY SOFTWARE THAT IS LICENSED TO THE INDIVIDUAL THAT ORIGINALLY OBTAINED IT FROM ORACLE.  IT MAY NOT BE TRANSFERRED OR DISTRIBUTED TO OTHER INDIVIDUALS, EVEN IF THOSE INDIVIDUALS ARE EMPLOYED BY THE SAME ENTITY.  IF YOU DID NOT OBTAIN THIS IMAGE FROM AN ORACLE WEBSITE OR FROM AN AUTHORIZED ORACLE EMPLOYEE OR CONTRACTOR, YOU MAY NOT USE IT; VISIT WWW.ORACLE.COM/OTN TO DOWNLOAD A COPY YOU MAY USE.   
+
+IF YOU RECEIVED THIS VIRTUAL MACHINE IMAGE DIRECTLY FROM ORACLE THEN BY CONTINUING TO INSTALL THE VIRTUAL MACHINE IMAGE YOU AGREE TO THE LICENSE TERMS AT http://www.oracle.com/technetwork/licenses/vm-license-2130072.html 

README.md

@@ -0,0 +1,97 @@
+Oracle Developer Day VM Builder
+===============================
+
+Welcome to the packer build for the Oracle Developer Day vm.  
+This will build a Oracle Linux server and install a 12.2 Oracle database on the machine.  Follow the download instructions to download the software to install and follow the build instructions to build the machine.  You will need around 30gb of space free to store the downloads and run the build.
+
+There are several directories which are used in the build
+
+1. put_files_here - place ALL downloaded software here
+2. output - The resultant VirtualBox OVA will be placed here
+3. upload - This directory has all the install scripts for the downloaded software.  
+
+Mandatory Downloads
+-------------------
+**These two downloads are mandatory**.  If they are not here, the build will not start.
+
+* OracleLinux-R7-U3-Server-x86_64-dvd.iso (needs to be this exactly md5 checked)
+* linux\*122*_database.zip (12201 has been verified - silent install may be incompatible with later versions)
+
+Optional Downloads
+--------------
+If these files are available in the **put_files_here** directory, the build will try and install them.  
+
+* jdk-8u\*-linux-x64.tar.gz - **Oracle JDK**
+* sqldeveloper-\*-no-jre.zip - **Oracle SQL Developer**
+* datamodeler-\*-no-jre.zip **Oracle SQLDeveloper Data Modeler**
+* sqlcl-\*-no-jre.zip - **Oracle SQLcl**
+* apex_5\*.zip - **Oracle Application Express**
+* ords\*.zip - **Oracle REST Data Services** 
+ 
+Oracle Internal Files
+---------------------
+These files may not be available publicly
+
+* demos.zip - **Oracle Hands on Labs demos**
+* reset_xmldbjson ** Reset XMLdb demos**
+* master.zip - **Oracle Sample Schema**
+* mozillablob.zip - **firefox .mozilla for (json viewer) plugins and or bookmarks**
+* storm.zip - **exp dump of non sensative material. For geo location optional demo.**
+
+Build Instructions
+------------------
+
+>**bash start.sh "http://and_proxy_as_first_arg_if_wanted" export**
+
+NOTE Proxy value will be in the logs from bash set -x stored in ~oracle/log 
+If you are inside a VPN, the proxy settings allow the VM to contact yum
+update servers. optional export to export the ova.
+
+![packer build](images/packerbuild.png)
+
+Build Structure
+--------------------
+
+1. 1updateLinux.sh PROGRESS: 1/4 1updateLinux.sh - first script run - from plain iso. Set up second drive. Yum Updates. Reboot
+2. 2afterFirstReboot.sh  PROGRESS: 2/4 2afterFirstReboot.sh - after first reboot - before any oracle database software installed. install virtual box guest additions.
+3. 3topLevel.sh PROGRESS: 3/4 3topLevel.sh - main kick off script e.g. yum and call subscripts
+4. 3_1installDbtoolClientTools.sh PROGRESS: 3_1/4 3_1installDbtoolClientTools.sh - install dbtools client tools
+5. 3_2installDatabase.sh PROGRESS: 3_2/4 3_2installDatabase.sh - install the database with silent install
+6. 3_3passwordDoNotExpire.sh PROGRESS: 3_3/4 3_3passwordDoNotExpire.sh - Database post install updates - e.g. passwords do not expire.
+7. 3_4enableGeoRaster.sh PROGRESS: 3_4/4 3_4enableGeoRaster.sh - post database install and patches (if required)
+8. 3_5unzipLabDemos.sh PROGRESS: 3_5/4 3_5unzipLabDemos.sh - demos (labs)
+9. 3_6apexInstall.sh PROGRESS: 3_6/4 3_6apexInstall.sh - apex install
+10. 3_7ORDSInstall.sh PROGRESS: 3_7/4 3_7ORDSInstall.sh - ords install
+11. 3_8setupDemos.sh PROGRESS: 3_8/4 3_8setupDemos.sh - set up demos and run reset scripts
+12. PROGRESS: END OF SCRIPT: 4/4 for tracing information see local file log.zip
+
+Timings:
+
+Minimum (Oracle Linux + Oracle Database)  
+Total time 60 mins (accounted for time + 7 mins rounding error / ignore <2 minute).  
+
+boot,8  
+yum update,14  
+reboot ,3  
+run database software install (not database build),3   
+dbca (database build),7  
+password and 32kvarchar2,6  
+shrink (fill disk with 0 for easy compression),3  
+1 x 3 minute reboot + shutdown + minor <1 minute stuff,10  
+
+Maximum (Including all optional extras APEX/ORDS/((internal) demos)  
+Total time 113.5 mins (Accounted for time + 10.5 mins rounding error / ignore <2 minute).  
+boot,9  
+yum update,14  
+reboot,3  
+run database software install (not database build),4  
+dbca (database build),8  
+password and 32kvarchar <should be same as minimum?>,18  
+apex install ,16  
+ords install ,4  
+set up demos and run demo reset,18  
+shrink (fill disk with 0 for easy compression),2  
+1 x 3 minute reboot + shutdown + minor <1 minute stuff - this is too long,9  
+
+Removing the shrink.sh if not exporting - fills disk with 0 for easy compression.  
+Run shrink.sh over ssh if subsequently exporting to ova (brings ova file down 60% in size to under 8Gb)

apacheLicense.txt

@@ -0,0 +1,13 @@
+Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

images/packerbuild.png

@@ -0,0 +1,13 @@
+Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

license.txt

@@ -0,0 +1,96 @@
+{
+"variables": {
+	"iso_downloaded": null,
+	"oracle_database_downloaded": null,
+        "proxy": "",
+	"skip_export": "true" 
+},
+    "provisioners": [
+	{
+	    "type": "file",
+	    "source": "upload",
+	    "destination": "/tmp"
+	},
+        {
+            "type": "file",
+            "source": "put_files_here",
+            "destination": "/tmp"
+        },
+	{
+	    "type": "shell",
+	    "inline": ["echo 'oracle' | sudo -S -E  bash -c 'mv /tmp/upload /tmp/1 && chmod -R 777 /tmp/1 && chown -R oracle /tmp/1 && /tmp/1/1updateLinux.sh {{user `proxy`}} && /sbin/shutdown -r +1'"]
+	},
+	{
+	    "pause_before": "200s",
+  	    "type": "shell",
+  	    "inline": ["echo not necessaryQQQ /home/oracle/bin/nuketimestamps>/dev/null && echo 'oracle'|sudo -S -E bash -c '/tmp/1/2afterFirstReboot.sh 2>&1 | tee /root/log$$ && /tmp/1/3topLevel.sh && /sbin/shutdown -r +1'"
+              ]
+	},
+	{
+           "pause_before": "200s",
+	    "type": "file",
+	    "source": "log.zip",
+	    "destination": "log.zip",
+	    "direction": "download"
+	},
+	{
+  	    "type": "shell",
+  	    "inline": ["rm log.zip"
+              ]
+	}
+    ],
+    "builders": [
+	{
+	    "type": "virtualbox-iso",
+      "ssh_skip_nat_mapping": true,
+      "ssh_host": "127.0.0.1",
+      "ssh_port": 2222,
+	    "disk_size": "30000",
+	    "guest_os_type": "Oracle_64",
+	    "hard_drive_interface": "sata",
+	    "boot_command": [
+		"<wait10>",
+		"<tab><wait5> text<wait> noverifyssl <wait>",
+		"ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/buildTimeConfig.cfg<enter><wait>"
+	    ],
+	    "vboxmanage": [
+   		[ "modifyvm", "{{.Name}}", "--memory", "4096" ],
+   		[ "modifyvm", "{{.Name}}", "--cpus", "2" ],
+		[ "createhd", "--filename", "Oracle DB Developer VMo_2.vdi", "--size", "30000"],
+		[ "storageattach",  "{{.Name}}", "--storagectl", "SATA Controller", "--port", "1", "--device", "0" ,"--type", "hdd", "--medium", "Oracle DB Developer VMo_2.vdi"],
+		[ "modifyvm", "{{.Name}}", "--clipboard", "bidirectional"],
+		[ "modifyvm", "{{.Name}}", "--draganddrop", "bidirectional"],
+		[ "modifyvm", "{{.Name}}", "--paravirtprovider", "kvm"],
+		[ "modifyvm", "{{.Name}}", "--natpf1", "net8,tcp,,1521,,1521"],
+		[ "modifyvm", "{{.Name}}", "--natpf1", "ssh,tcp,,2222,,22"],
+		[ "modifyvm", "{{.Name}}", "--natpf1", "apex8080,tcp,,8080,,8080"],
+		[ "modifyvm", "{{.Name}}", "--natpf1", "apex8081,tcp,,8081,,8081"],
+		[ "modifyvm", "{{.Name}}", "--audio", "none"],
+		[ "modifyvm", "{{.Name}}", "--vram", "128"]
+   		],
+   	    "vboxmanage_post": [
+   			[ "modifyvm", "{{.Name}}", "--memory", "2048" ],
+   		[ "modifyvm", "{{.Name}}", "--cpus", "1" ],
+		[ "storageattach", "{{.Name}}", "--storagectl", "IDE Controller", "--port", "1", "--device", "0", "--type", "dvddrive",  "--medium", "emptydrive"]
+  	    ],
+	    "guest_additions_mode": "upload",
+	    "iso_url": "OracleLinux-R7-U3-Server-x86_64-dvd.iso",
+	    "iso_checksum_type": "md5",
+	    "iso_checksum": "e44eff40f3a16ffa008285e5c52e1199",
+	    "format": "ova",
+	    "boot_wait": "10s",
+	    "ssh_username": "oracle",
+	    "ssh_password": "oracle",
+	    "ssh_wait_timeout": "2400s",
+	    "output_directory": "output",
+	    "shutdown_command": "echo 'oracle' | sudo -S shutdown -P now",
+	    "post_shutdown_delay": "30s",
+	    "keep_registered": "{{user `skip_export`}}",
+	"skip_export": "{{user `skip_export`}}",
+		"shutdown_timeout": "10m",
+		"vm_name": "Oracle DB Developer VM",
+		"http_directory":"upload",
+	    "export_opts": ["--vsys", "0","--eulafile","license.txt", "--product", "Oracle RDBMS 12.2, Application Express 5.1,REST Data Services 3.0,SQL Developer 4.2", "--producturl", "http://otn.oracle.com", "--vendor", "Oracle", "--vendorurl", "http://www.oracle.com", "--version", "May_2017"]
+	}
+    ]
+}

packerConfig.json

@@ -0,0 +1,23 @@
+**Put files required in build here**
+
+Required Files
+--------------
+* OracleLinux-R7-U3-Server-x86_64-dvd.iso (needs to be this exactly md5 checked)
+* linux\*122*_database.zip (12201 has been verified - silent install may be incompatible with later versions)
+
+Optional Downloads
+--------------
+If these files are available in the **put_files_here** directory, the build will try and install them.  
+
+* demos.zip - **Oracle Hands on Labs demos**
+* jdk-8u\*-linux-x64.tar.gz - **Oracle JDK**
+* sqldeveloper-\*-no-jre.zip - **Oracle SQL Developer**
+* sqlcl-\*-no-jre.zip - **Oracle SQLcl**
+* apex_5\*.zip - **Oracle Application Express**
+* ords\*.zip - **Oracle REST Data Services** 
+* datamodeler-\*-no-jre.zip **Oracle SQLDeveloper Data Modeler**
+* mozillablob.zip - **firefox .mozilla for (json viewer) plugins and or bookmarks**
+* storm.zip - **exp dump of non sensative material. For geo location optional demo.**
+* reset_xmldbjson
+* master.zip - **Oracle Sample Schema**
+

put_files_here/README.md

@@ -0,0 +1,78 @@
+#!/bin/bash
+
+#
+# Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved. 
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License. 
+# 
+
+
+
+#################################################################################
+#
+# File: start.sh
+#
+# Description: Start script to check initial files required for starting the
+#              packer build.  At a minimum, the Linux iso and the database zip
+#              are required before the packer build will start.  Everything else
+#              is optional.
+#              if needed pass proxy in, 
+#              if needed use the string 'export' to produce an .ova file 
+#              (proxy or export in any order)
+#################################################################################
+
+#
+# Cleanup old builds and remove previous output if it exists.
+#
+function cleanup {
+	rm -rf *.vdi
+	rm -rf output_last
+	mv output output_last
+}
+
+#
+# Check that at aminimum, the Linux iso, and the main database install zip
+# are present otherwise, there is no point in continuing
+#
+function check_required_files {
+        export varsaccepted=
+        if test -f put_files_here/OracleLinux-R7-U3-Server-x86_64-dvd.iso                                            
+        then                                                                                                         
+                export varsaccepted=" -var iso_downloaded=y "
+                cp -n put_files_here/OracleLinux-R7-U3-Server-x86_64-dvd.iso .                                       
+        fi
+	if test -f put_files_here/linuxx64*122*database.zip
+	then
+		export varsaccepted=" $varsaccepted -var oracle_database_downloaded=y " 
+	fi
+	if test "m$2" = "mexport"
+	then
+	        export varsaccepted=" $varsaccepted -var proxy=$1 -var skip_export=false "
+	elif test "m$1" = "mexport"
+        then
+		export varsaccepted=" $varsaccepted -var proxy=$2 -var skip_export=false "
+	else
+		export varsaccepted=" $varsaccepted -var proxy=$1 "
+	fi
+}
+#
+# start the packer build with a proxy if it is set.
+#
+function start_packer_build {
+	packer build $varsaccepted packerConfig.json
+}
+
+#Lets go.
+cleanup
+check_required_files $1 $2
+start_packer_build 

start.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+
+#
+# Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved. 
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License. 
+# 
+
+#################################################################################
+#
+# File: 1updateLinux.sh
+#
+# Description: Start script to check initial files required for starting the
+#              packer build.  At a minimum, the Linux iso and the database zip
+#              are required before the packer build will start.  Everything else
+#              is optional.
+#              if needed pass proxy in as $1 
+#################################################################################
+
+function bootstrap_logging {
+#
+# Copying files to home which decorate logs, ie starting and ending phases.
+# buildTimeReportSkippingFile.sh is used when some install zips are not available and buildTimeEndOfAllScripts.sh
+# is the final end which will cleanup
+#
+cp /tmp/1/runTimeStartScript.sh /tmp/1/buildTimeEnd.sh /tmp/1/runTimeStagesText.txt /tmp/1/buildTimeEndOfAllScripts.sh /tmp/1/buildTimeReportSkippingFile.sh ~oracle
+touch ~oracle/debug_output.txt
+chmod 777 ~oracle/runTimeStartScript.sh ~oracle/buildTimeEnd.sh ~oracle/runTimeStagesText.txt ~oracle/debug_output.txt ~oracle/buildTimeReportSkippingFile.sh ~oracle/buildTimeEndOfAllScripts.sh
+chown oracle  ~oracle/runTimeStartScript.sh ~oracle/buildTimeEnd.sh ~oracle/runTimeStagesText.txt ~oracle/debug_output.txt ~oracle/buildTimeReportSkippingFile.sh ~oracle/buildTimeEndOfAllScripts.sh
+
+mkdir ~oracle/log
+chmod 777 ~oracle/log
+chown oracle ~oracle/log
+}
+
+#
+# Update linux with YUM
+#
+function update_linux {
+(/tmp/1/1updateLinux_inner.sh "$@" 2>&1) |  tee -a  ~oracle/log/thestdoutlog_`basename $0`_$$ | egrep '^PROGRESS'
+}
+
+bootstrap_logging
+update_linux "$@"