Case Study
Test suite audit and optimization for LCvista
When LCvista needed to improve the efficiency and reliability of its test suite, Lincoln Loop provided a phased solution that enhanced performance, reduced costs, and empowered the development team through training and ongoing support.
Background
LCvista, a SaaS provider of accounting compliance solutions, required a streamlined test suite to support its agile development process. Faced with long test runtimes, flakiness*, and limited trust in CI results, LCvista partnered with Lincoln Loop to enhance its testing capabilities. Following an initial audit, Lincoln Loop provided recommendations that LCvista adopted over multiple phases.
* Flakiness in tests refers to inconsistent test results, where the same test may pass or fail unpredictably due to external factors rather than actual issues in the code.
Challenge
LCvista’s existing test suite presented several critical obstacles that hindered its development team. With CI build times stretching to 75 minutes, developers struggled to trust the results, as intermittent test failures made it unclear whether issues were genuine or simply flakiness. These long wait times and unreliable feedback led the team to occasionally bypass CI builds, slowing deployment and increasing the risk of bugs reaching production.
Key challenges included:
- Database Bottlenecks: Thousands of database migrations caused significant delays, particularly in parallel test runs.
- Lack of Test Isolation: Tests interfered with one another, especially when accessing shared resources, making results unpredictable.
- Limited Coverage Visibility: Without a reliable metric for test coverage, the team lacked insight into the quality and completeness of their testing.
This combination of factors underscored the need for a comprehensive optimization and reliability overhaul to support faster, more reliable releases.
Client
Solution
Lincoln Loop addressed LCvista’s testing challenges with a phased approach, starting with an in-depth audit to identify bottlenecks and failures. This informed a targeted optimization plan that streamlined database configuration, minimizing setup times and eliminating repetitive migrations to accelerate test initialization.
The next step was to replace the standard Django test runner with Pytest, a robust testing framework that offered better tooling. This change required meticulous refactoring to ensure test isolation, and resolving issues where tests impacted each other’s results, especially in file operations. We isolated file-dependent tests, addressed leaky global variables, and implemented parallelization strategies, enabling LCvista to achieve consistent, trustworthy test outcomes.
To support LCvista’s long-term success, Lincoln Loop completed the engagement with extensive documentation and live training. This training focused on key methodologies, including best practices for isolating tests, working with external services, and ensuring reliable outcomes across development environments. By providing custom-tailored guidance and a Q&A session, Lincoln Loop empowered LCvista’s developers to maintain and extend their testing improvements independently, establishing a sustainable, high-quality testing culture.
The Right Fit for the Job
Our expertise in Django, test optimization, and building mission-critical software platforms positioned us as an ideal partner for LCvista. Our pragmatic, phased approach allowed LCvista to progressively adopt and refine best practices while keeping in line with internal constraints and priorities.
For the past month, our tests have consistently returned green check marks. When there's a red X, we know it’s due to a real issue. This has been a game changer.
Joe LCvista Team Member
Results
- CI Time Reduction: Improved test suite build time by 87%.
- Cost Savings: Estimated annual savings of $28,000 in testing costs.
- Developer Efficiency: 20 developers gained consistent, reliable CI feedback, allowing faster deployments.
- Sustainable Testing Culture: LCvista’s team was empowered to independently maintain high-quality testing standards through training and documentation.
Looking Forward
With a strong testing foundation, LCvista can maintain and evolve its testing practices to support new features and improvements. The training and resources provided by Lincoln Loop ensure that the development team can uphold these practices for long-term success.
By optimizing its approach to testing, LCvista is also moving forward with a more cost-effective test suite that balances performance and affordability. The training and resources provided by Lincoln Loop equip the development team to uphold these practices for long-term success, fostering sustainability and efficiency in their workflow.
$28,000
Estimated annual savings in testing costs
Let’s build something great
Our work has helped over
1 billion people,
delivering solutions that make a difference.
15+ years of working together,
expertise you can rely on
— no revolving doors.
