Software / Web
KareerHub approached Scaffold Digital to design, build and strategise how to best crawl, archive and make searchable over 100,000+ pages of content on a regular basis to help provide school-goers with up-to-date information centrally on courses and apprenticeships across the UK combined with labour market information.
To build a system that made it efficient to crawl a lot of content, categorise it and make it searchable to users with relevancy markers to judge the most appropriate content to show.
- Build a crawler engine
- Relate the data to searches
- Implement a SAAS model for licensing
After some initial research into the best platform and methods to use to create the crawler engine a specification for the system was drawn up and agreed with an AGILE methodology use to run this project.
Our UX/UI team got to work on designing the interface to be easy to work with and above all, easy to consume information within.
At the development stage we utilised Laravel to create a bespoke platform before using Docker as a deployment method ensuring ease of scale for crawler or web traffic related processes.