Intelligent Web Crawling for Financial Intelligence and Business Contact Discovery
Overview
A leading provider of financial information, investment research, and business intelligence needed an automated way to collect publicly available corporate, shareholder, regulatory, and executive contact information from across the web. Investment research, corporate intelligence, business development, and market analysis teams all depended on this information being accurate and current — but gathering it manually from regulatory filings, corporate websites, educational institutions, non-profits, law firms, and business directories was repetitive, slow, and hard to sustain given how constantly this kind of online content changes.
TechTiera engineers designed and implemented an intelligent web crawling platform on Tungsten Automation (formerly Kofax Kapow) to automate the discovery, extraction, validation, and consolidation of this publicly available information — enabling continuous monitoring across multiple data sources and significantly speeding up the availability of structured business intelligence while cutting manual research effort.
The Challenge
The client’s difficulty wasn’t finding this information once — it was doing it repeatedly, consistently, and at scale across sources that never stayed the same for long. Specifically:
- Corporate and shareholder information was dispersed across numerous regulatory filings, corporate websites, and other public sources
- Manually identifying major shareholders and executive contacts was time-consuming and prone to inconsistency
- Regulatory filings required continuous monitoring to catch newly published information as it appeared
- Different websites used different technologies and page structures, complicating extraction across the board
- Business users needed accurate, timely contact information to support research, relationship management, and market intelligence work
- Doing this at scale demanded an automated, maintainable solution — not more analysts doing the same manual searches
By the time TechTiera got involved, the volume and breadth of sources the client needed to track had outpaced what manual research could realistically sustain.
Solution
Solution Architecture
TechTiera engineers built a centralized intelligent web crawling framework capable of collecting structured information from multiple public information sources and regulatory repositories, combining generic reusable crawlers with domain-specific robots for the collection, validation, and storage of corporate intelligence:
The split between “generic” and “domain-specific” robots was deliberate — a lot of the underlying crawling logic could be shared across sources, while each sector (regulatory filings, shareholder data, business directories) still needed its own extraction rules tuned to that source’s particular structure.
Solution Components
Regulatory Filing Extraction
TechTiera engineers built generic web crawlers to monitor and extract information from publicly available N-PX (Proxy Voting) filings, capturing shareholder and investment-related data for downstream analysis. The automation monitored filing repositories, identified newly published filings, extracted structured information, and normalized it for enterprise use — catching new filings as they appeared rather than on a periodic manual check.
Shareholder Intelligence Automation
Specialized robots collected information on major shareholders of publicly listed financial institutions across Nordic markets — Finland, Sweden, and Denmark — extracting major shareholder details, ownership information, corporate relationships, and publicly available investment disclosures.
Intelligent Business Contact Discovery
TechTiera engineers built generic search-based crawlers that leveraged publicly available search results to identify professional contact information associated with organizations — professional profiles, executive names, organizational roles, business contact references, and corporate affiliations. This was designed from the outset to collect only publicly available information, in line with applicable organizational policies and legal requirements.
Industry-Specific Contact Intelligence
Beyond the generic crawlers, TechTiera engineers built specialized robots tuned to collect publicly available organizational and contact information across educational institutions, non-profit organizations, law firms, public companies, and private equity firms — each robot optimized for the particular structure and navigation pattern of its target sites, since a one-size-fits-all approach doesn’t hold up well across sectors this different.
Centralized Data Repository
Extracted information was validated and consolidated into a centralized SQL Server repository, supporting historical tracking, searchable business intelligence, data normalization, duplicate detection, and structured reporting.
Deployment & Managed Services
Following implementation, TechTiera continued providing production monitoring, bot maintenance, enhancement development, website change management, incident management, performance optimization, and continuous managed services.
4. TechTiera Responsibilities
TechTiera delivered the complete project lifecycle, including:
- Business requirements analysis
- Solution architecture & technical design
- Customer workshops
- Generic web crawler development
- Industry-specific robot development
- Regulatory filing automation
- Data validation
- Database integration
- Testing and validation
- Production deployment
- Production monitoring
- Managed services and continuous improvement
5. Technology Stack
| RPA Platform: | Tungsten RPA (Kofax Kapow) 9.7.3 / 9.7.4 / 10.2 |
| Development Environment: | Kapow Design Studio |
| Database: | Microsoft SQL Server 2008 |
| Web Technologies: | HTML5 |
| Data Acquisition: | Intelligent Web Crawling |
| Data Sources: | Public Regulatory Filings, Corporate Websites, Public Business Directories |
| Search Integration: | Search Engine-Based Discovery |
| Operating Environment: | Windows |
| Managed Services: | Production Bot Monitoring & Support |
6. Results and Business Impact
Measurable outcomes
| Area | Before | After |
|---|---|---|
| Regulatory filing monitoring | Manual, periodic review | Continuous automated monitoring of N-PX and other filings |
| Shareholder research | Manual, market by market | Automated across Nordic financial institutions |
| Contact discovery | Manual search per organization | Automated across thousands of organizations |
| Data quality | Inconsistent, manually compiled | Validated, normalized, deduplicated centrally |
| Scalability | Constrained by analyst capacity | Scales to expanding source coverage |
Business Benefits
Benefits achieved by TechTiera’s automation implementation:
7. ROI
Major outcomes for engagement include:
- Return on investment within 6–12 months
- Significant reduction in manual research and information gathering costs
- Faster delivery of actionable business intelligence
- Improved productivity for research and investment analysis teams
- Greater coverage of publicly available information sources without increasing headcount
8. Project Execution Timeline
9. Project Hurdles and Lessons Learned
Key Challenges
Public websites and regulatory repositories came in wildly diverse layouts, navigation structures, and underlying technologies, which meant crawler designs had to be adaptable rather than built against a single assumed structure. Keeping extraction logic accurate across multiple sectors and geographies required components that were reusable and configurable, not sector-specific one-offs repeated over and over. Website structures changed frequently, which meant ongoing maintenance and proactive monitoring were part of the operating model from day one, not an afterthought. Getting to genuinely high-quality, deduplicated business intelligence required validation and normalization logic robust enough to catch inconsistencies across very different source types. And scaling to a growing number of monitored organizations took careful performance optimization as coverage expanded.
Lessons Learned
A generic crawler framework significantly cut development effort every time a new information source needed to be onboarded — new sources rarely meant starting over. Modular robot architecture made the whole system more adaptable as website structures shifted underneath it, which happened often. Centralized validation and normalization turned out to be essential for producing business intelligence people could actually trust and act on. Continuous monitoring and managed services were what kept performance sustained over time rather than degrading quietly. And collecting and managing publicly available information within clear governance frameworks — knowing exactly what’s in scope and what isn’t — improved both data quality and operational reliability rather than being just a compliance checkbox.
10. Best Practices Implemented
- Modular and reusable web crawler architecture
- Configurable extraction templates for different industry sectors
- Automated validation and duplicate detection
- Centralized monitoring and production support
- Exception handling with configurable retry mechanisms
- Comprehensive audit logging and operational reporting
- Standardized deployment and release management processes
- Proactive maintenance to accommodate changes in external websites and regulatory repositories
- Scalable framework supporting rapid onboarding of additional information sources
11. Security & Compliance
The solution was designed in accordance with enterprise governance and security practices, including:
- Collection of publicly available information only, in alignment with applicable organizational policies and legal requirements
- Secure management of automation infrastructure and production environments
- Role-based access controls for bot administration and monitoring
- Comprehensive audit trails for automation execution and data processing
- Version-controlled development and deployment of automation assets
- Controlled change management for enhancements and production updates
- Compliance with enterprise information security, operational governance, and quality management standards
12. Business Value Delivered
TechTiera delivered an enterprise-scale business intelligence automation platform that transformed the client’s ability to collect and maintain publicly available corporate, shareholder, and organizational information. By automating the extraction of regulatory filings, shareholder disclosures, and business contact data across multiple industries and geographies, the solution significantly reduced manual research, improved information quality, and accelerated the availability of actionable market intelligence. Through end-to-end delivery — architecture, development, deployment, and ongoing managed services — TechTiera established a scalable and resilient automation framework capable of supporting evolving business intelligence requirements while enabling more informed strategic, investment, and business development decisions.