Page History: How do I scrape the viewstate or other hidden variable?
Compare Page Revisions
Page Revision: 2009/06/10 18:08
Scraping the viewstate is tricky in two ways, it's hidden and it's long.
The following steps assumes you've turned the wizards off in Tools menu->Options
It also assumes you have Web Scraper 5.5.1 and that you've executed the script in
this article.
- To scrape a hidden variable you need to scrape a visible one first, then change the field tags to get the hidden one. To make sure you get the tags right though, the page should be downloaded in a package first, as specified in the how to build a package article.
- Then create a new dataset by selecting some random text on the page and press Ctrl+d to add a new dataset
- Choose the lowest number
- Now click the Edit(Advanced) link at the bottom of the window that pops up and enter a unique string on the page that occurs before the viewstate, <HTML usually works
- Click the Database tab and select Local SQL Instance, write in the Database and table names, and click OK
- Select some random text on the page, preferably before the viewsate (in case other things need to be scraped, selecting the correct position helps)
- Select the text in the lower pane and press Ctrl+f to add a new field
- In the database tab, change the Column Length to -1 so that the field is represented as a VARCHAR(MAX) in the database
- Now click the Tags(Manual) tab and check the Enable Manual Override box and enter the following for viewstate
- Start Tag:
id="__VIEWSTATE" value="
- End Tag:
"
- If you're not doing viewstate, do a test extraction and look at click the Step-By-Step Replay at the bottom, then look at the HTML and see what your start and end tags should be
- Now press play to do a test extraction