Update 2.0 – Upgraded Screen Capture & Scraping

Posted on: November 27th, 2019 by Tyler Puig in Updates

We’ve made some major changes to URL Profiler’s Screen Capture and scraping features.

Improved: Screen Capture

We have now implemented Chromium, which is essentially a headless browser that, in use, is the same as Google Chrome. We were previously using PhantomJS.

This means that our screen capture feature is now more accurate and should work more effectively – especially with sites that are Javascript heavy.

Added: Chrome Rendering

Most of URL Profiler’s features rely on data scraping. By default, the software will use the raw HTML (similar to clicking “View Source” on your browser). Then, it will search for certain elements and scrape the content itself.

We have now added the feature/option of having the page HTML rendered before scraping. Thus, all the JavaScript will load before URL Profiler searches for the element to scrape.

One thing to note is that for a large majority of sites, JavaScript is not actually changing the page content – so you would not see any difference scraping source HTML or rendered HTML.

However, there are an increasing number of websites that employ JavaScript frameworks such as React or AngularJS. On these sites, JavaScript basically changes all of the page content, so it is impossible to scrape the content without rendering first.

If you’d like to switch to Chrome rendering, go to Settings, then click the Rendering tab.

Once changed from Text to Chrome, URL Profiler will now use Chrome to render the HTML of the page before it scrapes anything.

The new Chrome rendering function will have an effect on the following features:

Note: The rendering option will be set to Text by default, even when restarting or re-opening the application.

Bug Fixes

  • Fixed screen capture export issue on Mac machines.
  • Fixed export issue that would add a double period for the root domain.
  • Fixed Pagespeed column alignment when the API fails.

Downloads

Existing customers or existing trial users can grab the new update from here:

If you’ve not tried URL Profiler yet, you can start a free 14 day trial here. The trial is full featured, and you don’t need to give us any payment details to get started.