Perl/CPAN Weather::Underground: Could not retrieve HTML

[peter_b]

[PROBLEM]
I'm using "wxer.pl" to retrieve weather information (temperature, humidity) for monitoring purposes.
It uses CPAN module "Weather::Underground to fetch the actual data.
All of a sudden, it stopped working, throwing the following error message:

Error, calling get_weather() failed: Could not retrieve HTML document http://mobile.wunderground.com/cgi-bin/ ... ery=$PLACE

[SOLUTION]
Install the CPAN module "LWP::Protocol::https" in order to access SSL secured websites.

In recent Debian-based distros, it could/should be in the repositories under the following name:

Code: Select all

liblwp-protocol-https-perl

You could also install it using "cpan":

Code: Select all

$ sudo cpan install LWP::Protocol::https

[REMARKS]
The same code worked flawless on another machine, and I also could "wget" the URL listed as not working.
I've manually tried to reproduce the call of "LWP::Simple->get()", in order to gain more details about what's going wrong.

The result was:

LWP will support https URLs if the LWP::Protocol::https module is installed.

Opening the site "http://mobile.wunderground.com", I now saw that it automatically redirects to https://

Installing the "LWP::Protocol::https" CPAN Module was not as straightforward as expected, because "SSLeay" failed to compile - because "libssl-dev" package was missing.

After that, the "LWP::Protocol::https" didn't install, and threw the following error:

# Failed test at t/apache.t line 15.
#
# Failed test at t/apache.t line 16.
# 'Can't connect to www.apache.org:443 (certificate+verify failed)

I couldn't figure out why, but in order to proceed somehow, I force-installed the CPAN module:

Code: Select all

$ cpan -fi LWP::Protocol::https

To be continued...

[peter_b]

In my case, I disabled the SSL host verification, due to missing SSL root certificates.

I did this, by adding the following line to Underground.pm

Code: Select all

$ua->ssl_opts(verify_hostname=>0);

This makes it possible to retrieve data over SSL/https, even if the certificate is untrusted (e.g. self-signed).
Of course, this is not the encouraged way of doing things, but in my case I thought it's okay to access the weather data

I put the above mentioned line in front of the code where the document URL is actually loaded:

Code: Select all

$ua->ssl_opts(verify_hostname=>0);

    $oldagent = $ua->agent();
    $ua->agent("Weather::Underground version $VERSION");
    $document = get($self->{_url});
    $ua->agent($oldagent);

When I have time, and I find a better solution, I'll update this post...