Web Scraper and Web Macros FAQs

Page History: How to Build a Package


Compare Page Revisions



« Older Revision - Back to Page History - Newer Revision »


Page Revision: 2009/05/13 18:54


This method of building a package has been developed after building hundreds of packages. It uses a template that can be found here:

Navigation:

The idea is to test navigation to each type of page on the site before worrying about extracting the data.
  • Insert the URL or POST that navigates to the first page you need to extract into the template Package->Steps tab->Listings 1st Page->File/Form List. This URL is typically to a top level page like first page of listings, top level category page, or home page/search page that gets the cookie needed to navigate the rest of the site.
  • Run the package and double click the URL in the window that pops up. Make sure the downloaded file has the data you need. If not, try creating a step before this one that navigates to the home page or blank search page so that a cookie for the site can be obtained. If that doesn't work, you may need to change some HTTP headers in Package->Steps tab->YourStepName->Advanced tab->Http Client. Use an HTTP sniffer to find out what these should be. Cookie then Referral URL then and User Agent are the most important.
  • Repeat this process for sample URL's of other types of pages you need to navigate down to, like Detail pages. Try to see if you can get to the detail pages directly by figuring out a pattern in the URL's. This can often save a lot of time.
*
PoweredBy
Create a Page | Administration | File Management | Login/Logout | Language Selection | Your Profile |Create Account