{"id":1434,"date":"2017-08-01T15:08:16","date_gmt":"2017-08-01T15:08:16","guid":{"rendered":"http:\/\/press3.mcs.anl.gov\/scienceclouds\/?page_id=1434"},"modified":"2017-08-01T15:08:16","modified_gmt":"2017-08-01T15:08:16","slug":"cloud-trace-format","status":"publish","type":"page","link":"https:\/\/wordpress.cels.anl.gov\/scienceclouds\/cloud-traces\/cloud-trace-format\/","title":{"rendered":"Cloud trace format"},"content":{"rendered":"\n

Version: 0.3<\/h4>\n\n\n\n

Date: November 1, 2019<\/p>\n\n\n\n

Major Changes<\/strong><\/p>\n\n\n\n

We adapted the trace format from the last version, so that the new format can be used for both virtual machine and bare-metal instances. <\/span><\/p>\n\n\n\n

Anonymization Techniques<\/strong><\/p>\n\n\n\n

(No change from the last version)<\/em><\/p>\n\n\n\n

For confidentiality reasons, we have anonymized certain fields in the traces. Different anonymization methods were applied to different fields.<\/span><\/p>\n\n\n\n

Hashed<\/em>: We applied keyed cryptographic hash to the fields INSTANCE_UUID, INSTANCE_NAME, USER_ID, PROJECT_ID and HOST_NAME (PHYSICAL). The same key and hashed method were applied to the HOST_NAME (PHYSICAL) fields in both tables, so the two tables can reference each other. <\/span><\/p>\n\n\n\n

Ordered<\/em>: This technique is applied to RACK field in the machine events table. The list of observed unique RACK values is sorted. Then, we assigned sequential numbers starting with 0 to the items of the RACK list, and the observed values are mapped onto these numbers.<\/span><\/p>\n\n\n\n

Time and Timestamps<\/strong><\/p>\n\n\n\n

(No change from the last version)<\/em><\/p>\n\n\n\n

For instance events, we provided two times — the event start time and the event finish time, but for machine events, we only provide one time — the event time. We present all time fields in two ways. The actual time in UTC and the time difference in seconds between the actual time and the trace period start time. For the event times that were before the trace period start time, we marked the time difference field -1.<\/span><\/p>\n\n\n\n

Data Tables<\/strong><\/p>\n\n\n\n

Machine Events<\/p>\n\n\n\n

File Name: machine_events.csv<\/p>\n\n\n\n

\n\n\n\n\n\n\n\n\n
Trace Field Name<\/strong><\/td>\nDatabase Fields<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
EVENT_TIME<\/span><\/td>\n\nvm<\/span><\/strong>
\nnova.compute_nodes.created_at<\/span>
\nnova.compute_nodes.updated_at<\/span>
\nnova.compute_nodes.deleted_at<\/span>
\nnova.services.updated_at<\/span>
\nbare-metal<\/span><\/strong>
\nironic.nodes.created_at<\/span>
\nblazar.computehosts.updated_at<\/span>
\nblazar.computehost_extra_capabilities
.created_at<\/span>
\nblazar.computehost_extra_capabilities.updated_at<\/span>
\n<\/td>\n		The time when the event happened in UTC.<\/span><\/td>\n<\/tr>\n
EVENT_TIME_SEC<\/span><\/td>\nN\/A<\/span><\/td>\nThe time difference in seconds between the EVENT_TIME and the trace epoch time.<\/span><\/td>\n<\/tr>\n
HOST_NAME (PHYSICAL)<\/span><\/td>\n\nvm<\/span><\/strong>
\nnova.compute_nodes.host<\/span>
\nbare-metal<\/span><\/strong>
\nironic.nodes.uuid<\/span>\n<\/td>\n		Name\/ID of the physical machine. Reference the instance events table using this field.<\/span><\/td>\n<\/tr>\n
EVENT<\/span><\/td>\nN\/A<\/span><\/td>\nCREATE, UPDATE, DELETE, DISABLE, ENABLE<\/span><\/td>\n<\/tr>\n
PROPERTIES<\/span><\/td>\n\nvm<\/span><\/strong>
\ncompute_nodes.vcpus<\/span>
\ncompute_nodes.memory_mb<\/span>
\ncompute_nodes.local_gb<\/span>
\nbare-metal<\/span><\/strong>
\nblazar.computehost_extra_capcabilities.capability_name<\/span>
\nblazar.computehost_extra_capcabilities.capability_value<\/span>\n<\/td>\n		The properties of the machine. (JSON formatted filed.)<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n\n\n\n

<\/p>\n\n\n\n

Instance Events<\/p>\n\n\n\n

File Name: instance_events.csv<\/p>\n\n\n\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Trace Field Name<\/strong><\/td>\nNova Database Fields<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
INSTANCE_UUID<\/span><\/td>\ninstances.uuid<\/span><\/td>\nThe unique id for each instance created by OpenStack.<\/span><\/td>\n<\/tr>\n
EVENT<\/span><\/td>\ninstance_actions_events.event<\/span><\/td>\nThe event corresponds to the type of action performed on an OpenStack instance.<\/span><\/td>\n<\/tr>\n
START_TIME<\/span><\/td>\ninstance_actions_events.start_time<\/span><\/td>\nThe time when a related event started executing.<\/span><\/td>\n<\/tr>\n
START_SEC<\/span><\/td>\nN\/A<\/span><\/td>\nThe time difference, in seconds, between the START_TIME and the trace epoch time.<\/span><\/td>\n<\/tr>\n
FINISH_TIME<\/span><\/td>\ninstance_actions_events.finish_time<\/span><\/td>\nThe time when a related event completed.<\/span><\/td>\n<\/tr>\n
FINISH_SEC<\/span><\/td>\nN\/A<\/span><\/td>\nThe time difference, in seconds, between the FINISH_TIME and the trace epoch time.<\/span><\/td>\n<\/tr>\n
EVENT_DURATION<\/span><\/td>\nN\/A<\/span><\/td>\nThe time difference between START_TIME and FINISH_TIME.<\/span><\/td>\n<\/tr>\n
RESULT<\/span><\/td>\ninstance_actions_events.result<\/span><\/td>\nStatus of each action event. If the result is not success or null, then this action event was not successfully.<\/span><\/td>\n<\/tr>\n
INSTANCE_NAME<\/span><\/td>\ninstances.hostname<\/span><\/td>\nThe name provided by the user while creating the instance. This name is not unique. Many instances can have the same instance name.<\/span><\/td>\n<\/tr>\n
USER_ID<\/span><\/td>\ninstances.user_id<\/span><\/td>\nThe user identifier of the account from which the instance was created.<\/span><\/td>\n<\/tr>\n
PROJECT_ID<\/span><\/td>\ninstances.project_id<\/span><\/td>\nThe project identifier of the account from which the instance was created.<\/span><\/td>\n<\/tr>\n
HOST_NAME (PHYSICAL)<\/span><\/td>\ninstances.host<\/span><\/td>\nName\/ID of the physical computer hosting the instance. Reference the machine events table using this field.<\/span><\/td>\n<\/tr>\n
PROPERTIES<\/span><\/td>\n\nvm<\/span><\/strong>
\ninstances.memory_mb<\/span>
\ninstances.root_gb<\/span>
\ninstances.vcpus<\/span>
\nbare-metal<\/span><\/strong>
\nN\/A<\/span>\n<\/td>\n		The properties of the instance. (JSON formatted field.)<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n\n\n\n
\n\n\n\n

Version: 0.2<\/h4>\n\n\n\n

Date: December 17, 2018<\/p>\n\n\n\n

Major Changes<\/strong><\/p>\n\n\n\n

Along with the instance events table, a new machine events table is added to the cloud traces. Each machine (physical host) is described by one or more records in the machine events table. <\/span><\/p>\n\n\n\n

Anonymization Techniques<\/strong><\/p>\n\n\n\n

For confidentiality reasons, we have anonymized certain fields in the traces. Different anonymization methods were applied to different fields.<\/span><\/p>\n\n\n\n

Hashed<\/em>: We applied keyed cryptographic hash to the fields INSTANCE_UUID, INSTANCE_NAME, USER_ID, PROJECT_ID and HOST_NAME (PHYSICAL). The same key and hashed method were applied to the HOST_NAME (PHYSICAL) fields in both tables, so the two tables can reference each other. <\/span><\/p>\n\n\n\n

Ordered<\/em>: This technique is applied to RACK field in the machine events table. The list of observed unique RACK values is sorted. Then, we assigned sequential numbers starting with 0 to the items of the RACK list, and the observed values are mapped onto these numbers.<\/span><\/p>\n\n\n\n

Time and Timestamps<\/strong><\/p>\n\n\n\n

For instance events, we provided two times — the event start time and the event finish time, but for machine events, we only provide one time — the event time. We present all time fields in two ways. The actual time in UTC and the time difference in seconds between the actual time and the trace period start time. For the event times that were before the trace period start time, we marked the time difference field -1.<\/span><\/p>\n\n\n\n

Data Tables<\/strong><\/p>\n\n\n\n

Machine Events<\/p>\n\n\n\n

File Name: machine_events.csv<\/p>\n\n\n\n

\n\n\n\n\n\n\n\n\n\n\n\n
Trace Field Name<\/strong><\/td>\nNova Database Fields<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
EVENT_TIME<\/span><\/td>\ncompute_nodes.created_at<\/span>\ncompute_nodes.updated_at<\/span>\ncompute_nodes.deleted_at<\/span>\nservices.updated_at<\/span>\n<\/td>\nThe time when the event happened in UTC.<\/span><\/td>\n<\/tr>\n
EVENT_TIME_SEC<\/span><\/td>\nN\/A<\/span><\/td>\nThe time difference in seconds between the EVENT_TIME and the trace period start time.<\/span><\/td>\n<\/tr>\n
HOST_NAME (PHYSICAL)<\/span><\/td>\ncompute_nodes.host<\/span><\/td>\nName of the physical machine hosting the KVM instance. Reference the instance events table using this field.<\/span><\/td>\n<\/tr>\n
RACK<\/span><\/td>\nN\/A<\/span><\/td>\nThe rack number which the machine belongs to. Extracted from the host name.<\/span><\/td>\n<\/tr>\n
EVENT<\/span><\/td>\nN\/A<\/span><\/td>\nCREATE, UPDATE, DELETE, DISABLE, ENABLE<\/span><\/td>\n<\/tr>\n
VCPU_CAPACITY<\/span><\/td>\ncompute_nodes.vcpus<\/span><\/td>\nThe available number of virtual CPUs.<\/span><\/td>\n<\/tr>\n
MEMORY_CAPACITY_MB<\/span><\/td>\ncompute_nodes.memory_mb<\/span><\/td>\nThe available memory in MB.<\/span><\/td>\n<\/tr>\n
DISK_CAPACITY_GB<\/span><\/td>\ncompute_nodes.local_gb<\/span><\/td>\nThe available disk in GB.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n\n\n\n

<\/p>\n\n\n\n

Instance Events<\/p>\n\n\n\n

File Name: instance_events.csv<\/p>\n\n\n\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Trace Field Name<\/strong><\/td>\nNova Database Fields<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
INSTANCE_UUID<\/span><\/td>\ninstances.uuid<\/span><\/td>\nThe unique id for each KVM instance created by OpenStack.<\/span><\/td>\n<\/tr>\n
EVENT<\/span><\/td>\ninstance_actions_events.event<\/span><\/td>\nThe event corresponds to the type of action performed on an OpenStack instance.<\/span><\/td>\n<\/tr>\n
START_TIME<\/span><\/td>\ninstance_actions_events.start_time<\/span><\/td>\nThe time when a related event started executing.<\/span><\/td>\n<\/tr>\n
START_SEC<\/span><\/td>\nN\/A<\/span><\/td>\nThe time difference, in seconds, between the START_TIME and the trace period start time.<\/span><\/td>\n<\/tr>\n
FINISH_TIME<\/span><\/td>\ninstance_actions_events.finish_time<\/span><\/td>\nThe time when a related event completed.<\/span><\/td>\n<\/tr>\n
FINISH_SEC<\/span><\/td>\nN\/A<\/span><\/td>\nThe time difference, in seconds, between the FINISH_TIME and the trace period start time.<\/span><\/td>\n<\/tr>\n
EVENT_DURATION<\/span><\/td>\nN\/A<\/span><\/td>\nThe time difference between START_TIME and FINISH_TIME.<\/span><\/td>\n<\/tr>\n
RESULT<\/span><\/td>\ninstance_actions_events.result<\/span><\/td>\nStatus of each action event. If the result is not success or null, then this action event was not successfully.<\/span><\/td>\n<\/tr>\n
MEMORY_MB<\/span><\/td>\ninstances.memory_mb<\/span><\/td>\nAmount of RAM assigned to an instance at creation.<\/span><\/td>\n<\/tr>\n
DISK_GB<\/span><\/td>\ninstances.root_gb<\/span><\/td>\nAmount of disk assigned to an instance at creation.<\/span><\/td>\n<\/tr>\n
VCPUS<\/span><\/td>\ninstances.vcpus<\/span><\/td>\nThe number of virtual CPUs assigned to an instance at creation.<\/span><\/td>\n<\/tr>\n
INSTANCE_NAME<\/span><\/td>\ninstances.hostname<\/span><\/td>\nThe name provided by the user while creating the VM. This name is not unique. Many instances can have the same instance name.<\/span><\/td>\n<\/tr>\n
USER_ID<\/span><\/td>\ninstances.user_id<\/span><\/td>\nThe user identifier of the account from which the VM was created.<\/span><\/td>\n<\/tr>\n
PROJECT_ID<\/span><\/td>\ninstances.project_id<\/span><\/td>\nThe project identifier of the account from which the VM was created.<\/span><\/td>\n<\/tr>\n
HOST_NAME(PHYSICAL)<\/span><\/td>\ninstances.host<\/span><\/td>\nName of the physical computer hosting the KVM instance. Reference the machine events table using this field.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n\n\n\n
\n\n\n\n

Version: 0.1<\/h4>\n\n\n\n

Date: July 31, 2017<\/p>\n\n\n\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Trace field name<\/strong><\/td>\nNova database fields<\/strong><\/td>\n<\/tr>\n
INSTANCE_UUID<\/td>\ninstances.uuid<\/td>\n<\/tr>\n
EVENT<\/td>\ninstance_actions_events.event<\/td>\n<\/tr>\n
START_TIME<\/td>\ninstance_actions_events.start_time<\/td>\n<\/tr>\n
START_SEC<\/td>\nderived from START_TIME<\/em><\/td>\n<\/tr>\n
FINISH_TIME<\/td>\ninstance_actions_events.finish_time<\/td>\n<\/tr>\n
FINISH_SEC<\/td>\nderived from FINISH_TIME<\/em><\/td>\n<\/tr>\n
EVENT_DURATION<\/td>\nderived from START_TIME and FINISH_TIME<\/em><\/td>\n<\/tr>\n
RESULT<\/td>\ninstance_actions_events.result<\/td>\n<\/tr>\n
MEMORY_MB<\/td>\ninstances.memory_mb<\/td>\n<\/tr>\n
DISK_GB<\/td>\ninstances.root_gb<\/td>\n<\/tr>\n
VCPUS<\/td>\ninstances.vcpus<\/td>\n<\/tr>\n
INSTANCE_NAME<\/td>\ninstances.hostname<\/td>\n<\/tr>\n
USER_ID<\/td>\ninstances.user_id<\/td>\n<\/tr>\n
PROJECT_ID<\/td>\ninstances.project_id<\/td>\n<\/tr>\n
HOST_NAME(PHYSICAL)<\/td>\ninstances.host<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n\n\n\n

Details of each trace fields:<\/p>\n\n\n\n