Ever needed to rise up a Safety Operations Middle (SOC) in two days? That is the monumental problem confronted by Cisco engineers at numerous occasions and conferences across the globe all year long. Chances are you’ll ask, “How is it possible to deploy a full-fledged SOC with just two days of preparation?” The important thing to with the ability to make the almost unattainable occur is our customized “SOC in a Box”. It’s primarily a roadshow case, racked with the required {hardware} for a SOC, that may be packed and shipped to any location. On this weblog, I’ll undergo the phases of making ready the package from ideation in San Jose to implementation on the RSA Convention in San Francisco.
Section 1: Dusting off the cobwebs
Arriving on the Cisco campus in San Jose, California, and strolling into the lab on Monday morning one week earlier than RSAC was so nostalgic. It jogged my memory of my days as a TAC (Cisco Assist staff) engineer doing buyer “recreates” (recreate points reported by prospects) within the lab. What a sight to behold, a multi-story workplace constructing fully devoted to lab house!
Once we lastly discovered our gear, the case appeared dusty… prefer it hadn’t been touched in a 12 months (as a result of it hadn’t). Actually the case simply wanted a little bit tender loving care. We began with a drawing of what we wished to construct: Within the depiction the web cloud is definitely the Moscone Middle community and isn’t managed/secured by RSA
Most of this part concerned cleansing out the case, eradicating any pointless {hardware}, securing the remaining {hardware} with correct rackmounts and screws, and including zip ties for energy cable administration.
Subsequent, we would have liked to reimage the UCS C220 M5 and set up the ESXi 8.0, a strong, bare-metal hypervisor that installs immediately onto your bodily server. Right here is the place the hurdles start! After making a bootable USB thumb drive, we confronted a problem with the server not recognizing the drive. Shout out to Robert Harris for establishing CIMC and utilizing the browser based mostly KVM to add the ISO file.
With the server sorted, it was time to maneuver on to the change. After a “write erase” of the config, we seen the change solely had two 10G interfaces, one other hurdle as we would have liked a minimal of 4 10G interfaces. After lunch, we made a fast cease on the Cisco “repot depot” storefront in Constructing 9 to select up a “nm-4-10g” community module for the Catalyst 3850. After a little bit of networking Layer 1 troubleshooting, we realized the change was not recognizing the community module. We additionally tried to reimage the change from rommon and set up the most recent software program however that didn’t resolve the problem both. Shout out to Matt Vander Horst, who helped us clear this hurdle by trying up the spec sheet and discovering that the 24 port Catalyst 3850 doesn’t assist the 4x10G community module and that we would want a 48 port Catalyst 3850.
With the change on pause, we moved on to the Cisco Firepower 4125 Firewall. Within the RSAC SOC, we usually prefer to run the most recent and best software program releases so we will showcase the brand new options and put our Cisco safety instruments to the take a look at in a posh, real-world setting. This firewall wanted an FXOS improve to run FTD 7.4.1. Though FXOS 2.14 put in efficiently, we got here to the subsequent hurdle after we seen a fault with one in every of disks within the chassis. Dinkar Sharma helped us with the disk fault however, even after opening a TAC case and getting assist from Ravi Kiran Nagaraja, the problem continued. Shout out to Justin Murphy and Shannon Wellington for delivering an 800 GB SSD drive from their lab on quick discover as our last-ditch effort. With the brand new disk put in we crossed our fingers however to no avail. Once more, the identical error relating to a failure to format the disk which signifies a problem with the chassis itself.
At this level, our “SOC in a Box” might have been a failure. The transport deadline was approaching quick, and we didn’t have the mandatory change or a working Firewall. Discuss a significant hurdle!
Section 2: Beg, borrow, and steal (not likely, as a result of we requested properly)
After a easy alternate on Webex groups, Zohreh Kehzri got here to the rescue with a 48 port Catalyst 3850 with eight 10G ports! We walked over to constructing 17 (getting our steps in across the San Jose campus) to select up the 3850 and, another reimage later, we had a functioning change, lastly getting us over this hurdle. After the struggles of part 1, we have been glad to take a fast win. With the brand new change racked within the case, it was time to drop our homegrown unit off for transport earlier than we headed over to the Safety Summit. Here’s what our “SOC in a Box” appeared like proper earlier than we shipped it.
On the Safety Summit, I noticed Eric Kostlan, the resident firewall guru. Figuring out that we have been in determined want of a {hardware} firewall, I went again to the “beg, borrow, and steal” strategy, asking Erik if he might assist. In not-so-shocking vogue, he checked his lab setting and sourced a spare firewall. After listening to of the problems we confronted with the opposite chassis, he even made the additional effort to make sure FXOS 2.14 was put in efficiently and the safety engine got here up wholesome, getting us over another hurdle.
As soon as the periods on the Safety Summit have been over round 6:30 pm, we went to Eric’s lab and borrowed the firewall out of his racks earlier than heading to dinner. The subsequent day, I hoisted the brand new FTD 4115 into an Uber XL and headed to San Francisco to prepare for the convention. (A community engineer’s dream to Uber a firewall from metropolis to metropolis!)
Now that now we have acquired all of the parts of the puzzle, it’s time to place the items collectively.
Section 3: Energy it up, wire it up
On Saturday morning, Could 4, Moscone Middle in San Francisco was buzzing with convention preparation. It’s actually mindboggling to see the present ground rework from naked concrete to a accomplished showcase in 48 hours. I picked up my badge and wheeled the case over to the South Expo. Here’s what the case appeared like subsequent to the 10G fiber drop earlier than any arrange was began.
This part is usually powering up the {hardware} and wiring it with web entry, administration entry, and the SPAN (Switched Port Analyzer is a devoted port on a change that takes a mirrored copy of community visitors from throughout the change to be despatched to a vacation spot) from Moscone Community Operations Middle. Shout out to Ryan Maclennan for working with the on-site technicians to make sure Layer 1 on the 10G SPAN was working appropriately. The 24 port Catalyst 3850 was used for the SOC administration community, a subnet offered by the Moscone Middle. After re-IP-addressing the administration interfaces of all our gadgets, the idea of the community was on-line.
In these conditions, it’s crucial to be versatile. Since we have been unsure on learn how to change the IP addresses of the Cisco Telemetry Dealer (CTB) supervisor and CTB dealer node, we shortly pivoted the Observable Community Equipment (ONA), which might accomplish the identical objective of changing the SPAN to IPFIX (Web Protocol Circulate Info Export) to pump as much as Cisco XDR.
Moreover, we completed the Firewall logical gadget set up and linked the SPAN to a passive interface and accomplished the remainder of the fundamental configuration from the Cisco Safe Firewall Administration Middle (FMC). Subsequent, we put in Splunk Enterprise Safety (ES) on an Ubuntu machine and configured the Splunk Technical Add-ons (TAs) for Cisco XDR integration, eStreamer log ingestion, and Firewall dashboarding. Shout out to Seyed Khadem-Djahaghi for the customized darkish mode dashboard he created within the Splunk console.
Here’s what our customized “SOC in the Box” appeared like wired up and absolutely operational, linked to the Moscone NOC and NetWitness Platform. We have now room for NetWitness home equipment and their 140TB of storage for these community packets.
Section 4 – Huge time on the massive screens
With our “SOC in a Box” operational and all our instruments on-line, it was time for the ending touches of placing up the gorgeous dashboards on the massive. On Sunday afternoon, we have been capable of login to the Cisco Safety instruments and showcase them on the “SOC Dashboard” on public show between North and South Expo. At this level, it felt like we had efficiently completed the race and cleared all of the hurdles. Right here’s what it appeared like earlier than the present opened; Cisco Safe Cloud Analytics, Cisco XDR, Splunk ES, and FMC have been on the massive screens.
We had loads of guests throughout present hours analyzing the SOC Dashboard.
On Tuesday morning after we got here into the SOC, we bumped into that surprising remaining hurdle – the Splunk was down! After checking on the command line interface, we discovered that the disk was full – the 2TB we had initially allotted had been used. Fortunately, we had a spare UCS C240 M4 with 18TB of storage in our “SOC in a Box”, we borrowed a VGA monitor and USB keyboard from the RSA A/V staff so we might spin up the server on the fly and allocate extra storage to Splunk ES. Hurdle cleared, and we coasted to a profitable end.
Throughout our SOC excursions, we defined to the convention attendees (together with our very personal Engineering SVP, Shaila Shankar) how we’re utilizing our instruments for risk searching and incident response! (Above is one in every of many selfies I’ve taken with Shaila.)
Elements Used:
- Change: Catalyst 3850 (24 port)
- Change: Catalyst 3850 with 10G SFP+ (48 port)
- Firewall: Safe Firewall 4115
- Server: UCS C220 M5
- Server: UCS C240 M4
Within the topology proven above, the purple field encompasses our on-premises “SOC in a Box” infrastructure. Beginning within the backside proper, the Umbrella Digital home equipment are deployed throughout the Moscone Community Operations Middle. By assigning the digital home equipment because the DNS servers within the DHCP scope all DNS queries on the community are seen to Cisco Umbrella – Person Safety Suite.
Subsequent, the SPAN of all convention community visitors is plugged into the Catalyst 3850, which is actually getting used as a SPAN replicator. From the change, the SPAN visitors is shipped to a Safe Firewall 4115 in Intrusion Detection mode for deep packet inspection, an On-premises community equipment (ONA) to get IPFIX (Web Protocol Circulate Info Export) knowledge to XDR, and to NetWitness, the place the complete pcap (packet seize) is saved.
Firewall Administration Middle (FMC) makes use of eStreamer to ship detection and connection knowledge to Splunk and NetWitness. Information are despatched to Malware Analytics from each FMC and Netwitness. Cisco XDR integrates with Umbrella, Safe Firewall, Malware Analytics, NetWitness, Splunk, and quite a few risk intel sources for risk searching and incident response.
A brand new addition to our SOC this 12 months was Cisco Safe Entry. By deploying the useful resource connector in our ESXi, the on-premises gear is accessible from wherever offered correct authentication has taken place. Our customized “SOC in a Box” was one of many highlights of the SOC excursions and generated fairly a bit of pleasure round Cisco Safety!
So lengthy RSAC 2024!! We’ll be again once more subsequent 12 months!
To study extra:
Because of:
- Robert Harris
- Matt Vander Horst
- Dinkar Sharma
- Eric Kostlan
- Ryan Maclennan
- Seyed Khadem-Djahaghi
- The RSA Convention workers
- The Moscone Community Operations Middle
- And the complete Cisco and NetWitness RSAC SOC staff members
We’d love to listen to what you suppose. Ask a Query, Remark Under, and Keep Related with Cisco Safety on social!
Cisco Safety Social Channels
Instagram
Fb
Twitter
LinkedIn
Share: