As a Sales Engineer, a particular aspect of the job has no defined description. You can only do that part of the job by having solid experience, a strong will, and the ability to listen to a customer and act upon their needs. The required skills to complete a project like this are heavy discovery skills, internal and external negotiation skills, and the ability to see a customer’s needs from both tactical and strategic viewpoints. I was tasked with this project in mid-2021. I was tasked with this project in mid-2021. I was given no timeline for completion and resource limits other than, don’t spend any money.
The problem
Tulane University signed a 3-year contract with AT&T for Managed Threat Detection and Response at the end of 2020. For over six months, the deployment carried on, and the customer CIO was very upset that they were no closer to getting the deployment in place than on day one. They sent logs into the platform across three sensors and killed the control node daily. My job – fix it.
There have been several issues that have plagued the MTDR deployment over the last three months. Below is an outline of the issues. Sensor issues ingesting logs from the Palo Alto PA7050 Firewall. The overall volume of logs created and passed to the sensor causes the sensor to overload, peak its resources, and then crash.
With the overload of the PA 7050 Firewall-associated sensor, the number of logs passing through the sensor to the control node is causing overutilization of resources, which in turn causes the Cloud Based Control Node to falter. This is causing slow processing of the data as it is sent up to the cloud from the premise-based sensor.
Threat Model Workshop was not conducted at the onset of the deployment because of an issue with the contracting process and the time it took. The TMW has since been completed.
The Solution
Once the TMW was completed I worked with the deployment team and the customer to rerun the deployment process using the TMW results. I broke the deployment down into a multi-phase project to critical, essential, and less important assets to be migrated into the platform for monitoring.
The first asset we migrated into the platform was the PA7050 to prevent overloading the controller. I worked with the customer firewall engineers and showed them how to break down the logging into segments by log type and subnets to only bring in the logs for assets we wanted to monitor.
We then migrated in critical servers, load balancers, and critical switching. The logs from these assets were minuscule compared to the PA7050. They got the customer up to just beyond the 90% utilization point for the amount of storage acquired in the contract. At this point, the customer ran with the platform logging to ensure they would get usable, actionable data from their network. Once they did, we spoke with the CIO and rewrote the contract from 6TB of storage monthly to 20TB of monthly storage.
The customer was happy with the work we put in for them, and we proved viability and improvement in security efforts. Once the customer signed the new contract, the deployment team worked with the customer to migrate the remaining assets on the projects into the platform and, at the last conversation, had about 85% visibility across their entire networking, server, and end-point environments.
