Challenges, Benefits and Best Practices of Performance Focused DevOps 


The Presentation inside:

Slide 0

Wolfgang Gottesheim @gottesheim Challenges, Benefits and Best Practices of Performance Focused DevOps 


Slide 1


Slide 2

? ? ? Unit/Integration Tests Acceptance Tests Capacity Tests Release Developers When do we find performance problems?


Slide 3


Slide 4


Slide 5


Slide 6


Slide 7

Test


Slide 8

Production


Slide 9

Dev Ops


Slide 10

What Operations tells Developers…


Slide 11

What Developers would like to know


Slide 12

What Developers would like to know Top contributor is save of Mage_Core_Model_Abstract 70% of that time comes from Sales_Model_Quote Where are those calls coming from?


Slide 13


Slide 14


Slide 15

Because NOBODY wants this …


Slide 16


Slide 17

~80% of problems caused by ~20% patterns YES we know this 80% Dev Time in Bug Fixing $60B Defect Costs BUT


Slide 18

Define metrics that are understood across teams Share measurement methods and tools Make performance part of agile stories Broaden the View


Slide 19


Slide 20

But…


Slide 21

It’s a culture thing


Slide 22

Culture Measure Share Automation


Slide 23

Starting from… Production Environment Developers CI Server Testing Environment Release


Slide 24

…or maybe… Production Environment Developers CI Server Testing Environment Release


Slide 25

We want to get to… Commit Stage Automated Acceptance Testing Automated Capacity Testing Release Developers


Slide 26

METRICS to look at, WHY we want them, HOW they help


Slide 27

Queues and Pools


Slide 28

Online Banking: Slow Balance Check 1.69min (=101s!) To check balance! 87% spent in IIS 600! SQL Executions


Slide 29

#1: Time Spent in IIS? Thread 32 in IIS took 87s to pass control to Thread 30 in ASP.NET


Slide 30

#2: SQL Executions! A new connection for every statement…


Slide 31

#2: SQL Executions! continued … #1: Same SQL is executed 67! times #2: NO PREPARATION because everything executed on new Connection


Slide 32

Lessons Learned! ASP.NET Worker Thread Pool Sizing! DB Connection Pools More Efficient SQL


Slide 33

Helpful Metrics Idle vs. Busy Threads # SQLs / Request # Opened connections


Slide 34

Push without Plan


Slide 35

Mobile Landing Page of Super Bowl Ad 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB http://apmblog.dynatrace.com/2014/01/31 /technical-and-business-web-performance-tips-for-super-bowl-ad-landing-pages/


Slide 36

m.store.com redirects to www.store.com ALL CSS and JS files are redirected to the www domain This is a lot of time “wasted” especially on high latency mobile connections http://apmblog.dynatrace.com/2013/12/02/the-terrible-website-performance-mistakes-of-mobile-shopping-sites-in-2013/


Slide 37

Fifa.com during Worldcup http://apmblog.dynatrace.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/


Slide 38

Helpful Metrics Load Time # Images # Redirects # HTTP 3xx, 4xx, 5xx Size of resources


Slide 39

Reusing Components


Slide 40

Requirement: We need a report


Slide 41

Using Hibernate results in 4k+ SQL Statements to display 3 items! Hibernate Executes 4k+ Statements Individual Execution VERY FAST But Total SUM takes 6s http://apmblog.dynatrace.com/2014/04/23/database-access-quality-metrics-for-your-continuous-delivery-pipeline/


Slide 42

Helpful Metrics # SQL Statement Executions # of same SQLs Result Set Size


Slide 43

Tools and Frameworks


Slide 44

Online Bank: Transaction History CSV Download! Building CSV output in memory… Problem: Takes 207s! To download. 87% of Time spent in Garbage Collection


Slide 45

Online Store: Rendering Search Result Problem: 4.4s to render result page Root Cause: Custom RegEx Library with performance issues on large strings


Slide 46

Helpful Metrics Memory Usage Time Spent in APIs


Slide 47

Architectural Decisions


Slide 48

Symptoms HTML takes between 60 and 120s to render High GC Time Assumptions Bad GC Tuning Probably bad Database Performance as rendering was simple Project: Online Room Reservation System


Slide 49

Developers built own monitoring void roomreservationReport(int roomid) { long startTime = System.currentTimeMillis(); Object data = loadDataForRoom(roomid); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, roomid); } Result: Avg. Data Load Time: 45s! DB Tool says: Avg. SQL Query: <1ms!


Slide 50

#1: Loading too much data 24889! Calls to the Database API! High CPU and High Memory Usage to keep all data in Memory


Slide 51

#2: On individual connections 12444! individual connections Classical N+1 Query Problem Individual SQL really <1ms


Slide 52

#3: Putting all data in temp Hashtable Lots of time spent in Hashtable.get Called from their Entity Objects


Slide 53

Custom Measuring Was impacted by Garbage Collection Just measured overall time but not # SQL Executions Learn SQL and don’t use Hashtables as Workaround Lesson Learned void roomreservationReport(int roomid) { long startTime = System.currentTimeMillis(); Object data = loadDataForRoom(roomid); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, roomid); }


Slide 54

Helpful Metrics # SQL Executions # of SAME SQLs Connection Acquisition Time


Slide 55

Performance as a Quality Gate Automated collection of performance metrics in test runs Comparison of performance metrics across builds Automated analysis of performance metrics to identify outliers Automated notifications on performance issues in tests Measurements accessible and shareable across teams Actionable data through deep transactional insight Integration with build automation tools and practices


Slide 56

PERFORMANCE as part of our Continuous Delivery Process Commit Stage Automated Acceptance Testing Automated Capacity Testing Release Developers


Slide 57

# Images # Redirects Size of Resources # SQL Executions # of SAME SQLs # of Connections Time Spent in API Remember: Use Tools to measure… # Calls into API # Functional Errors 3rd Party calls # of Domains Total Page Size # Items per Page # AJAX per Page


Slide 58


Slide 59

Collaborate Verify Measure


Slide 60


Slide 61


Slide 62

If we do all that …


Slide 63

Unit/Integration Tests Acceptance Tests Capacity Tests Release Developers


Slide 64

CANCELLED


Slide 65

… so there is more time for …


Slide 66


Slide 67

blog.dynatrace.com bit.ly/dttrial


Slide 68

A FINAL THOUGHT!


Slide 69


Slide 70

Putting it into Test Automation We identified a regresesion Problem solved Exceptions probably reason for failed tests Now we have the functional and architectural confidence Let’s look behind the scenes


Slide 71

And in your Pipeline Commit Stage Compile Execute Unit Test Code Analysis Build installers Automated Acceptance Testing Automated Capacity Testing Manual testing Key showcases Exploratory testing Unit & Integration Tests Functional Tests Performance Tests Production Monitoring Functional Tests


Slide 72

Wolfgang Gottesheim Free Tools: http://bit.ly/dttrial Follow me @gottesheim Email me [email protected] http://blog.dynatrace.com


×

HTML:





Ссылка: