Netflix's Analytics and Developer Experience team investigated a UI latency issue in their Workbench product, a remote development environment based on Titus. The issue caused slowness in JupyterLab when certain notebooks were run. Initial suspects included the use of pystan and nest_asyncio, noisy container neighbors, and potential network problems. However, the root cause was traced to a JupyterLab extension called jupyter_resource_usage, which caused CPU contention due to its linear time complexity function for reading smaps_rollup files. Disabling this extension resolved the issue.

15m read timeFrom netflixtechblog.com
Post cover image
Table of contents
Investigation of a Workbench UI Latency IssueOverviewSymptomQuantify the SlownessBlame The NotebookBlame Noisy NeighborsBlame The NetworkThe Minimal ReproductionQuestionsMore CluesRoot Cause AnalysisMore Quantitative AnalysisSolutionSummary

Sort: