How To Scrape Data Off A Part Of A Site That Requires User Navigation

For example say I am trying to scrape from this page: http://www.swtor.com/leaderboards/pvp/solo It only shows top 50 results, pretty easy to go to this link and scrape those data.

Solution 1:

If you open the "Network" panel on you browser developer tools, you can see the XMLHttpRequest (XHR) requests the site does to load the table data:

http://www.swtor.com/lb/data?page=1&column=pvp_ranked_solo&season=6
http://www.swtor.com/lb/data?page=2&column=pvp_ranked_solo&season=6

This endpoint returns very handy JSONs and now it's just a matter of running as many requests as necessary. Tip: the number of pages is also on the returned JSON, so there's no need to parse a single HTML page even if you want all records.