Managing data
To ensure that you can securely manage your data when you use IBM Cloud Monitoring, it is important to know exactly what data is stored and encrypted, and how you can delete any stored personal data.
How your data is collected in IBM Cloud Monitoring
When you configure a Monitoring agent to collect and forward data to an Monitoring instance, data is automatically collected and available for analysis through the web UI. You can configure the Monitoring agent to connect to the Monitoring instance via the public network or the private network.
To connect to resources in your account over the IBM Cloud public network, you can configure an agent to send metrics by using a public endpoint. The environment where the agent is running requires internet access to use the public endpoint.
You can enable virtual routing and forwarding (VRF) to move IP routing for your account and all of its resources into a separate routing table. If VRF is enabled, you can then enable IBM Cloud service endpoints to connect directly to resources without using the public network. To configure an agent to send metrics by using a private endpoint, you must enable virtual routing and forwarding (VRF) for your account. Once the account is VRF enabled, the Monitoring agent can be configured to use the private network by using the Private Endpoint as the ingestion URL.
- Private endpoints are not accessible from the public internet.
- All traffic is routed to the IBM Cloud private network.
Monitoring agent data is collected at 10-seconds frequency. Data that is published by platform metrics is collected on a 1-minute frequency.
Captures
In IBM Cloud Monitoring, you can also enable captures when you configure a Monitoring agent. A capture is a trace file that you can use to analyze what happens in a host during a time frame.
- Captures contain system calls, and other OS events.
- You can enable or disable this feature per node when you configure the Monitoring agent that collects metrics from that node. A node can be a host, a container, a virtual machine, a bare metal, or any metrics source where you install a Monitoring agent.
By default, the IBM instructions for configuring the Monitoring agents disables the capture feature. If you choose to follow other instructions, IBM recommends disabling capture by setting the sysdig_capture_enabled: false in
your dragent.yaml for Linux installations or the Kubernetes sysdig-agent Deployment custom resource.
When Captures are enabled, to avoid a security incident and potentially exposing data outside of your organization, check your organization's security policies before you enable captures on a node. Consider disabling the Capture feature for all your Monitoring agents.
How your data is stored in IBM Cloud Monitoring
IBM Cloud Monitoring collects and aggregates metrics.
Data location
Metric data is hosted on the IBM Cloud.
- Each multi-zone region (MZR) location collects and aggregates metrics for each instance of the IBM Cloud Monitoring that runs in that location.
- Data is colocated in the region where the IBM Cloud Monitoring instance is provisioned. For example, metric data for an instance that is provisioned in US South is hosted in the US South region.
Data retention
Data is retained for each instance based on a roll-up policy.
As time progresses, the data is rolled up from a fine granularity to a coarser one by the end 2 months.
The roll-up policy describes the granularity of the data over time:
- Data is retained at 10-second resolution for the first 4 hours.
- Data is retained at 1-minute resolution for 2 days.
- Data is retained at 10-minute resolution for 2 weeks.
- Data is retained at 1-hour resolution for 2 months.
- Data is retained at 1-day resolution for 15 months.
Data availability
Data is available for a maximum of 15 months.
User metadata is always available.
After you remove a Monitoring agent from a host or container, historical data is not deleted.
- Data is available for a maximum of 15 months.
- Data is available for analysis through the web UI for the time period that the agent was installed and reporting.
After you delete an instance of the IBM Cloud Monitoring service, data is not available for search and analysis.
Deleting your data
Deleting metric data
Metric data is deleted automatically after 15 months.
Deleting user metadata
User metadata, such as alerts, dashboards, teams, and users, is never deleted.
You must open a case through support to request the metadata to be deleted. For more information, see Creating support cases.
Deleting a subset of data
Deletion of a subset of data is not supported.
For example, deletion of data that is collected from 1 Monitoring agent in a Monitoring instance is not supported.
Deleting captures
When you delete a capture, the data file for that capture is automatically deleted.
Deleting an IBM Cloud Monitoring instance
When you delete an instance of IBM Cloud Monitoring from the IBM Cloud, you must open a case through support to request the data to be deleted. For more information, see contacting support.