![]() The attributed flow data drives various use cases within Netflix like network monitoring and network usage forecasting available via Lumen dashboards and machine learning based network segmentation. The attributed flows are pushed to Keystone that routes them to the Hive and Druid datastores. It performs real time attribution of flow data with application metadata from Sonar. Sonar is an IPv6 and IPv4 address identity tracking service.įlow Collector consumes two data streams, the IP address change events from Sonar via Kafka and eBPF flow log data from the Flow Exporter sidecars. We use Sonar to attribute an IP address to a specific application at a particular time. IP addresses within the cloud can move from one EC2 instance or Titus container to another over time. So how do we ingest and enrich these flows at scale ?įlow Collector is a regional service that ingests and enriches flows. ![]() These metrics are visualized using Lumen, a self-service dashboarding infrastructure. The Flow Exporter also publishes various operational metrics to Atlas. The runtime behavior of the Flow Exporter can be dynamically managed by configuration changes via Fast Properties. The choice of transport protocols like GRPC, HTTPS & UDP is runtime dependent on characteristics of the instance placement. ![]() The sidecar has been implemented by leveraging the highly performant eBPF along with carefully chosen transport protocols to consume less than 1% of CPU and memory on any instance in our fleet. In some ways, eBPF does to the kernel what JavaScript does to websites: it allows all sorts of new applications to be created.Īn eBPF flow log record represents one or more network flows that contain TCP/IP statistics that occur within a variable aggregation interval. What is BPF?īerkeley Packet Filter (BPF) is an in-kernel execution engine that processes a virtual instruction set, and has been extended as eBPF for providing a safe way to extend kernel functionality. The Flow Exporter is a sidecar that uses eBPF tracepoints to capture TCP flows at near real time on instances that power the Netflix microservices architecture. By collecting, accessing and analyzing network data from a variety of sources like VPC Flow Logs, ELB Access Logs, eBPF flow logs on the instances, etc, we can provide network insight to users and central teams through multiple data visualization techniques like Lumen, Atlas, etc. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |