#! /usr/bin/env perl# Planet Jupiter is a feed aggregator that creates a single HTML file# Copyright (C) 2020–2021 Alex Schroeder <alex@gnu.org># This program is free software: you can redistribute it and/or modify# it under the terms of the GNU Affero General Public License as published by# the Free Software Foundation, either version 3 of the License, or# (at your option) any later version.## This program is distributed in the hope that it will be useful,# but WITHOUT ANY WARRANTY; without even the implied warranty of# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the# GNU Affero General Public License for more details.## You should have received a copy of the GNU Affero General Public License# along with this program. If not, see <https://www.gnu.org/licenses/>.packageJupiter;useEncode::Locale;useEncode;useutf8;binmode(STDOUT,":utf8");binmode(STDERR,":utf8");=encoding UTF8=head1 NAMEjupiter - turn a list of feeds into a HTML page, a river of news=head1 SYNOPSISTo update the feeds from one or more OPML files:B<jupiter update> I<feed.opml> … [I</regex/> …]To generate F<index.html>:B<jupiter html> I<feed.opml>=head1 DESCRIPTIONPlanet Jupiter is used to pull together the latest updates from a bunch of othersites and display them on a single web page, the "river of news". The sites weget our updates from are defined in an OPML file.A river of news, according to Dave Winer, is a feed aggregator. New items appearat the top and old items disappear at the bottom. When it's gone, it's gone.There is no count of unread items. The goal is to fight the I<fear of missingout> (FOMO).Each item looks similar to every other: headline, link, an extract, maybe a dateand an author. Extracts contain but the beginning of the article's text; allmarkup is removed; no images. The goal is to make the page easy to skim.Scroll down until you find something interesting and follow the link to theoriginal article if you want to read it.=head2 The OPML fileYou B<need> an OPML file. It's an XML file linking to I<feeds>. Here's anexample listing just one feed. In order to add more, add more C<outline>elements with the C<xmlUrl> attribute. The exact order and nesting does notmatter. People can I<import> these OPML files into their own feed readers andthus it may make sense to spend a bit more effort in making it presentable.<opml version="2.0"><body><outline title="Alex Schroeder"xmlUrl="https://alexschroeder.ch/view/index.rss"/></body></opml>=head2 Update the feeds in your cacheThis is how you update the feeds in a file called C<feed.opml>. It downloads allthe feeds linked to in the OPML file and stores them in the cache directory.jupiter update feed.opmlThe directory used to keep a copy of all the feeds in the OPML file has the samename as the OPML file but without the .opml extension. In other words, if yourOPML file is called C<feed.opml> then the cache directory is called C<feed>.This operation takes long because it requests an update from all the siteslisted in your OPML file. Don't run it too often or you'll annoy the siteowners.The OPML file must use the .opml extension. You can update the feeds formultiple OPML files in one go.=head2 Adding just one feedAfter a while, the list of feeds in your OPML starts getting unwieldy. When youadd a new feed, you might not want to fetch all of them. In this case, provide aregular expression surrounded by slashes to the C<update> command:jupiter update feed.opml /example/Assuming a feed with a URL or title that matches the regular expression islisted in your OPML file, only that feed is going to get updated.There is no need to escape slashes in the regular expression: C<//rss/> worksjust fine. Beware shell escaping, however. Most likely, you need to surround theregular expression with single quotes if it contains spaces:jupiter update feed.opml '/Halberds & Helmets/'Notice how we assume that named entities such as C<&> have already beenparsed into the appropriate strings.=head2 Generate the HTMLThis is how you generate the C<index.html> file based on the feeds of yourC<feed.opml>. It assumes that you have already updated all the feeds (seeabove).jupiter html feed.opmlSee L</OPTIONS> for ways to change how the HTML is generated.=head2 Generate the RSS feedThis happens at the same time as when you generate the HTML. It takes all theentries that are being added to the HTML and puts the into a feed.See L</OPTIONS> for ways to change how the HTML is generated.=head2 Why separate the two steps?The first reason is that tinkering with the templates involves running theprogram again and again, and you don't want to contact all the sites wheneveryou update your templates.The other reason is that it allows you to create subsets. For example, you canfetch the feeds for three different OPML files:jupiter update osr.opml indie.opml other.opmlAnd then you can create three different HTML files:jupiter html osr.html osr.opmljupiter html indie.html indie.opmljupiter html rpg.html osr.opml indie.opml other.opmlFor an example of how it might look, check out the setup for the planets I run.=head2 What about the JSON file?There's a JSON file that gets generated and updated as you run Planet Jupiter.It's name depends on the OPML files used. It records metadata for every feed inthe OPML file that isn't stored in the feeds themselves.C<message> is the HTTP status message, or a similar message such as "No entrynewer than 90 days." This is set when updating the feeds in your cache.C<message> is the HTTP status code; this code could be the real status code fromthe server (such as 404 for a "not found" status) or one generated by Jupitersuch that it matches the status message (such as 206 for a "partial content"status when there aren't any recent entries in the feed). This is set whenupdating the feeds in your cache.C<title> is the site's title. When you update the feeds in your cache, it istaken from the OPML file. That's how the feed can have a title even if thedownload failed. When you generate the HTML, the feeds in the cache are parsedand if a title is provided, it is stored in the JSON file and overrides thetitle in the OPML file.C<link> is the site's link for humans. When you generate the HTML, the feeds inthe cache are parsed and if a link is provided, it is stored in the JSON file.If the OPML element contained a C<htmlURL> attribute, however, that takesprecedence. The reasoning is that when a podcast is hosted on a platform whichgenerates a link that you don't like and you know the link to the human-readableblog elsewhere, use the C<htmlURL> attribute in the OPML file to override this.C<last_modified> and C<etag> are two headers used for caching from the HTTPresponse that cannot be changed by data in the feed.If we run into problems downloading a feed, this setup allows us to still linkto the feeds that aren't working, using their correct names, and describing theerror we encountered.=head2 LoggingUse the C<--log=LEVEL> to set the log level. Valid values for LEVEL are debug,info, warn, error, and fatal.=head1 LICENSEGNU Affero General Public License=head1 INSTALLATIONUsing C<cpan>:cpan App::jupiterManual install:perl Makefile.PLmakemake install=head2 DependenciesTo run Jupiter on Debian we need:C<libmodern-perl-perl> for L<Modern::Perl>C<libmojolicious-perl> for L<Mojo::Template>, L<Mojo::UserAgent>, L<Mojo::Log>,L<Mojo::JSON>, and L<Mojo::Util>C<libxml-libxml-perl> for L<XML::LibXML>C<libfile-slurper-perl> for L<File::Slurper>C<libdatetime-perl> for L<DateTime>C<libdatetime-format-mail-perl> for L<DateTime::Format::Mail>C<libdatetime-format-iso8601-perl> for L<DateTime::Format::ISO8601>Unfortunately, L<Mojo::UserAgent::Role::Queued> isn't packaged for Debian.Therefore, let's build it and install it as a Debian package.sudo apt-get install libmodule-build-tiny-perlsudo apt-get install dh-make-perlsudo dh-make-perl --build --cpan Mojo::UserAgent::Role::Queueddpkg --install libmojo-useragent-role-queued-perl_1.15-1_all.debTo generate the C<README.md> from the source file, you need F<pod2markdown>which you get in C<libpod-markdown-perl>.=head1 FILESThere are a number of files in the F<share> directory which you can use asstarting points.F<template.html> is the HTML template.F<default.css> is a small CSS file used by F<template.html>.F<personalize.js> is a small Javascript file used by F<template.html> used toallow visitors to jump from one article to the next using C<J> and C<K>.F<jupiter.png> is used by F<template.html> as the icon.F<jupiter.svg> is used by F<template.html> as the logo.F<feed.png> is used by F<template.html> as the icon for the feeds in thesidebar.F<feed.rss> is the feed template.=head1 OPTIONSHTML generation uses a template, C<template.html>. It is written forC<Mojo::Template> and you can find it in the F<share> directory of yourdistribution. The default templates use other files, such as the logo, the feedicon, a CSS file, and a small Javascript snippet to enable navigation using theC<J> and C<K> keys (see above).You can specify a different HTML file to generate:B<jupiter html> I<your.html feed.opml>If you specify two HTML files, the first is the HTML file to generate and thesecond is the template to use. Both must use the C<.html> extension.B<jupiter html> I<your.html your-template.html feed.opml>Feed generation uses a template, C<feed.rss>. It writes all the entries into afile called C<feed.xml>. Again, the template is written for C<Mojo::Template>.You can specify up to two XML, RSS or ATOM files. They must uses one of thesethree extensions: C<.xml>, C<.rss>, or C<.atom>. The first is the name of thefeed to generate, the second is the template to use:B<jupiter html> I<atom.xml template.xml planet.html template.html feed.opml>In the above case, Planet Jupiter will write a feed called F<atom.xml> based onF<template.xml> and a HTML file called F<planet.html> based on F<template.html>,using the cached entries matching the feeds in F<feed.opml>.=cutuseDateTime;useFile::Basename;useModern::Perl;useMojo::Log;useMojo::Template;useMojo::UserAgent;usePod::Simple::Text;useXML::LibXML;our$log= Mojo::Log->new;my$xpc= XML::LibXML::XPathContext->new;my$undefined_date= DateTime->from_epoch(epoch=> 0 );my(%wday,%month,$wday_re,$month_re);%wday=qw (lun.Mon mar. Tue mer. Wed jeu. Thu ven. Fri sam. Sat dim. Sun);%month=qw (janv.Jan févr. Feb mars Mar avr. Apr mai May juin Junjuil. Jul août Aug sept. Sepoct. Oct nov. Nov déc. Dec);$wday_re=join('|',map{quotemeta}keys%wday)unless$wday_re;$month_re=join('|',map{quotemeta}keys%month)unless$month_re;# Our tests don't want to call main__PACKAGE__->mainunlesscaller;submain {my($log_level) =grep/^--log=/,@ARGV;$log->level(substr($log_level, 6))if$log_level;my($command) =grep/^[a-z]+$/,@ARGV;$command||='help';if($commandeq'update') {update_cache(@ARGV);}elsif($commandeq'html') {make_html(@ARGV);}else{my$parser= Pod::Simple::Text->new();$parser->parse_file($0);}}subupdate_cache {my($feeds,$files) = read_opml(@_);make_directories($feeds);load_feed_metadata($feeds,$files);my$ua= Mojo::UserAgent->new->with_roles('+Queued')->max_redirects(3)->max_active(5);make_promises($ua,$feeds);fetch_feeds($feeds);save_feed_metadata($feeds,$files);cleanup_cache($feeds);}submake_promises {my$ua=shift;my$feeds=shift;formy$feed(@$feeds) {my$url= html_unescape$feed->{url};# undo xml_escape for the request$ua->on(start=>sub{my($ua,$tx) =@_;$tx->req->headers->if_none_match($feed->{etag})if($feed->{etag});$tx->req->headers->if_modified_since($feed->{last_modified})if($feed->{last_modified});});$feed->{promise} =$ua->get_p($url)->catch(sub{$feed->{message} ="@_";$feed->{code} = 521;# returning 0 in the case of an error is important0; })# sleeping to stop blogger.com from blocking us->finally(sub{$log->debug($url);sleep2; });}}subfetch_feeds {my$feeds=shift;$log->info("Fetching feeds...");Mojo::Promise->all(map{$_->{promise} }@$feeds)->then(sub{# all returns the values in the same order!for(my$i= 0;$i<@_;$i++) {my$feed=$feeds->[$i];my$value=$_[$i];my$tx=$value->[0];# relies on catch returning 0 abovenextunless$tx;$feed->{message} =$tx->result->message;$feed->{code} =$tx->result->code;$feed->{last_modified} =$tx->result->headers->last_modified;$feed->{etag} =$tx->result->headers->etag;# save raw bytes if this is a successeval{ write_binary($feed->{cache_file},$tx->result->body) }if$tx->result->is_success;warn"Unable to write $feed->{cache_file}: $@\n"if$@;}})->catch(sub{warn"Something went wrong: @_";})->wait;}subload_feed_metadata {my$feeds=shift;my$files=shift;formy$file(@$files) {my$filename="$file->{path}/$file->{name}";nextunless-r"$filename.json";my$data= decode_json read_binary("$filename.json");formy$feed(@$feeds) {my$url=$feed->{url};nextunless$url;# don't overwrite title and htmlUrl from OPML file$feed->{title} =$data->{$url}->{title}if$data->{$url}->{title};$feed->{link} ||=$data->{$url}->{link};# all the other metadata is loaded from the JSON file$feed->{message} =$data->{$url}->{message};$feed->{code} =$data->{$url}->{code};$feed->{last_modified} =$data->{$url}->{last_modified};$feed->{etag} =$data->{$url}->{etag};}grep{$_->{opml_file} eq$file->{file} }@$feeds;}}subsave_feed_metadata {my$feeds=shift;my$files=shift;formy$file(@$files) {my$name=$file->{name};my%messages=map{my$feed=$_;$feed->{url} => {map{$_=>$feed->{$_} }grep{$feed->{$_} }qw(title link message code last_modified etag)};}grep{$_->{opml_file} eq$file->{file} }@$feeds;write_binary("$file->{path}/$file->{name}.json", encode_json \%messages);}}subcleanup_cache {my$feeds=shift;my%good=map{$_=> 1 } @{cache_files($feeds)};my@unused=grep{ not$good{$_} } @{existing_files($feeds)};if(@unused) {$log->info("Removing unused files from the cache...");foreach(@unused) {$log->info($_) }unlink@unused;}}subexisting_files {my$feeds=shift;my@files;formy$dir(uniqmap{$_->{cache_dir} }@$feeds) {push(@files, <"$dir/*">)if$dirand -d$dir;}return\@files;}subcache_files {my$feeds=shift;my@files=map{$_->{cache_file} }@$feeds;return\@files;}submake_directories {my$feeds=shift;formy$dir(uniqmap{$_->{cache_dir} }@$feeds) {if($dirand not -d$dir) {mkdir$dir;}}}submake_html {my($feeds,$files) = read_opml(@_);load_feed_metadata($feeds,$files);# load messages and codes for feedsmy$globals= globals($files);my$entries= entries($feeds, 4);# set data for feeds, tooadd_data($feeds,$entries);# extract data from the xmlsave_feed_metadata($feeds,$files);# save title and link for feeds$entries= limit($entries, 100);write_text(html_file(@_), apply_template(read_text(html_template_file(@_)),$globals,$feeds,$entries));write_text(feed_file(@_), apply_template(read_text(feed_template_file(@_)),$globals,$feeds,$entries));}subhtml_file {my($html) =grep/\.html$/,@_;return$html||'index.html';}subhtml_template_file {my($html,$template) =grep/\.html$/,@_;$template||= dist_file('App-jupiter','template.html');die"HTML template $template not found\n"unless-r$template;return$template;}subfeed_file {my($feed) =grep/\.(xml|rss|atom)$/,@_;return$feedif$feed;return'feed.xml';}subfeed_template_file {my($feed,$template) =grep/\.(xml|rss|atom)$/,@_;return$templateif$template;returndist_file('App-jupiter','feed.rss');}subapply_template {my$mnt= Mojo::Template->new;return$mnt->render(@_);}=head1 TEMPLATESThe page template is called with three hash references: C<globals>, C<feeds>,and C<entries>. The keys of these three hash references are documented below.The values of these hashes are all I<escaped HTML> except where noted (dates andfile names, for example).The technical details of how to write the templates are documented in the manpage for L<Mojo::Template>.=head2 GlobalsThere are not many global keys.B<date> is the the publication date of the HTML page, in ISO date format:YYYY-MM-DD.B<files> is the list of OPML files used.=cutsubglobals {my$files=shift;my@time=gmtime;my$today= DateTime->now->ymd;return{date=>$today,files=>$files};}=head2 Writing templates for feedsFeeds have the following keys available:B<title> is the title of the feed.B<url> is the URL of the feed (RSS or Atom). This is not the link to the site!B<link> is the URL of the web page (HTML). This is the link to the site.B<opml_file> is the file name where this feed is listed.B<cache_dir> is the directory where this feed is cached.B<message> is the HTTP status message or other warning or error that we gotwhile fetching the feed.B<code> is the HTTP status code we got while fetching the feed.B<doc> is the L<XML::LibXML::Document>. Could be either Atom or RSS!=cut# Creates list of feeds. Each feed is a hash with keys title, url, opml_file,# cache_dir and cache_file.subread_opml {my(@feeds,@files);my@filters=map{ decode(locale=>substr($_, 1, -1)) }grep/^\/.*\/$/,@_;formy$file(grep/\.opml$/,@_) {my$doc= XML::LibXML->load_xml(location=>$file);# this better have no errors!my@nodes=$doc->findnodes('//outline[./@xmlUrl]');my($name,$path) = fileparse($file,'.opml','.xml');formy$node(@nodes) {my$title= xml_escape$node->getAttribute('title');my$url= xml_escape$node->getAttribute('xmlUrl');nextif@filters> 0 and notgrep{$url=~ /$_/ or$title=~ /$_/ }@filters;my$link= xml_escape$node->getAttribute('htmlUrl');push@feeds, {title=>$title,# title in the OPML fileurl=>$url,# feed URL in the OPML filelink=>$link,# web URL in the OPML fileopml_file=>$file,cache_dir=>"$path/$name",cache_file=>"$path/$name/". slugify($url),};}warn"No feeds found in the OPML file $file\n"unless@nodes;push@files, {file=>$file,path=>$path,name=>$name};}@feeds= shuffle@feeds;return\@feeds, \@files;}subentries {my$feeds=shift;my$limit=shift;my$date= DateTime->now(time_zone=>'UTC')->subtract(days=> 90 );# compute oncemy$now= DateTime->now(time_zone=>'UTC');my@entries;formy$feed(@$feeds) {nextunless-r$feed->{cache_file};my$doc=eval{ XML::LibXML->load_xml(recover=> 2,location=>$feed->{cache_file} )};if(not$doc) {$feed->{message} = xml_escape"Parsing error: $@";$feed->{code} = 422;# unprocessablenext;}$feed->{doc} =$doc;my@nodes=$xpc->findnodes("/rss/channel/item | /atom:feed/atom:entry",$doc);if(not@nodes) {$feed->{message} ="Empty feed";$feed->{code} = 204;# no contentnext;}# if this is an Atom feed, we need to sort the entries ourselves (older entries at the end)my@candidates=map{my$entry= {};$entry->{element} =$_;$entry->{id} = id($_);$entry->{date} = updated($_) ||$undefined_date;$entry;}@nodes;@candidates=grep{ DateTime->compare($_->{date},$now) <= 0 }@candidates;@candidates= unique(sort{ DateTime->compare($b->{date},$a->{date} ) }@candidates);@candidates=@candidates[0 .. min($#candidates,$limit- 1)];# now that we have limited the candidates, let's add more metadata from the feedformy$entry(@candidates) {$entry->{feed} =$feed;# these two are already escaped$entry->{blog_title} =$feed->{title};$entry->{blog_url} =$feed->{url};}add_age_warning($feed, \@candidates,$date);push@entries,@candidates;}return\@entries;}subadd_age_warning {my$feed=shift;my$entries=shift;my$date=shift;# feed modification date is smaller than the date givenmy($node) =$xpc->findnodes("/rss/channel | /atom:feed",$feed->{doc});my$feed_date= updated($node);if($feed_dateand DateTime->compare($feed_date,$date) == -1) {$feed->{message} ="No feed updates in 90 days";$feed->{code} = 206;# partial contentreturn;}else{# or no entry found with a modification date equal or bigger than the date givenformy$entry(@$entries) {returnifDateTime->compare($entry->{date},$date) >= 0;}$feed->{message} ="No entry newer than 90 days";$feed->{code} = 206;# partial content}}subupdated {my$node=shift;returnunless$node;my@nodes=$xpc->findnodes('pubDate | atom:published | atom:updated',$node) orreturn;my$date=$nodes[0]->textContent;my$dt=eval{ DateTime::Format::Mail->parse_datetime($date) }||eval{ DateTime::Format::ISO8601->parse_datetime($date) }||eval{ DateTime::Format::Mail->parse_datetime(french($date)) };return$dt;}subfrench {my$date=shift;$date=~ s/^($wday_re)/$wday{$1}/;$date=~ s/\b($month_re)/$month{$1}/;return$date;}subid {my$node=shift;returnunless$node;my$id=$xpc->findvalue('guid | atom:id',$node);# id is mandatory for Atom$id||=$node->findvalue('link');# one of the following three is mandatory for RSS$id||=$node->findvalue('title');$id||=$node->findvalue('description');return$id;}subunique {my%seen;my@unique;formy$node(@_) {nextif$seen{$node->{id}};$seen{$node->{id}} = 1;push(@unique,$node);}return@unique;}sublimit {my$entries=shift;my$limit=shift;# we want the most recent entries overall@$entries=sort{ DateTime->compare($b->{date},$a->{date} ) } unique(@$entries);return[@$entries[0 .. min($#$entries,$limit- 1)]];}=head2 Writing templates for entriesEntries have the following keys available:B<title> is the title of the post.B<link> is the URL to the post on the web (probably a HTML page).B<blog_title> is the title of the site.B<blog_link> is the URL for the site on the web (probably a HTML page).B<blog_url> is the URL for the site's feed (RSS or Atom).B<authors> are the authors (or the Dublin Core contributor), a list of strings.B<date> is the publication date, as a DateTime object.B<day> is the publication date, in ISO date format: YYYY-MM-DD, for the UTCtimezone. The UTC timezone is picked so that the day doesn't jump back and forthwhen sorting entries by date.B<content> is the full post content, as string or encoded HTML.B<excerpt> is the post content, limited to 500 characters, with paragraphseparators instead of HTML elements, as HTML. It is not encoded because the ideais that it only gets added to the HTML and not to the feed, and the HTML itcontains is very controlled (only the pilcrow sign inside a span to indicateparagraph breaks).B<categories> are the categories, a list of strings.B<element> is for internal use only. It contains the L<XML::LibXML::Element>object. This could be RSS or Atom!B<feed> is for internal use only. It's a reference to the feed this entrybelongs to.=cutsubadd_data {my$feeds=shift;my$entries=shift;# A note on the use of xml_escape: whenever we get data from the feed itself,# it needs to be escaped if it gets printed into the HTML. For example: the# feed contains a feed title of "Foo & Bar". findvalue returns "Foo &# Bar". When the template inserts the title, however, we want "Foo & Bar",# not "Foo & Bar". Thus: any text we get from the feed needs to be escaped# if there's a chance we're going to print it again.formy$feed(@$feeds) {nextunless$feed->{doc};# title and url in the feed overrides title and xmlUrl set in the OPML (XML already escaped)$feed->{title} = xml_escape($xpc->findvalue('/rss/channel/title | /atom:feed/atom:title',$feed->{doc})) ||$feed->{title} ||"";$feed->{url} = xml_escape($xpc->findvalue('/atom:feed/atom:link[@rel="self"]/@href',$feed->{doc})) ||$feed->{url} ||"";# link in the feed does not override htmlUrl in the OPML (XML already escaped)$feed->{link} =$feed->{link} || xml_escape($xpc->findvalue('/rss/channel/link | /atom:feed/atom:link[@rel="alternate"][@type="text/html"]/@href',$feed->{doc})) ||"";# if they just pasted a domain "foo" then "//foo" is a valid URL$feed->{link} ="//".$feed->{link}unless$feed->{link} =~ /\/\//;}formy$entry(@$entries) {# copy from the feed (XML is already escaped)$entry->{blog_link} =$entry->{feed}->{link};$entry->{blog_title} =$entry->{feed}->{title};$entry->{blog_url} =$entry->{feed}->{url};# parse the elementsmy$element=$entry->{element};$entry->{title} = xml_escape$xpc->findvalue('title | atom:title',$element) ||"Untitled";my@links=map{ xml_escape$_->to_literal }map{$xpc->findnodes($_,$element) }# sorted by preferences!qw(link atom:link[@rel="alternate"][@type="text/html"]/@href atom:link/@href);$entry->{link} =shift(@links) ||"";my@authors=map{ without_email(xml_escape strip_html($_->to_literal)) }$xpc->findnodes('author | atom:author/atom:name | atom:contributor/atom:name | dc:creator | dc:contributor',$element);@authors=map{ without_email(xml_escape strip_html($_->to_literal)) }$xpc->findnodes('/atom:feed/atom:author/atom:name | '.'/atom:feed/atom:contributor/atom:name | '.'/rss/channel/dc:creator | '.'/rss/channel/dc:contributor | '.'/rss/channel/webMaster ',$element)unless@authors;$entry->{authors} =@authors? \@authors:undef;# key must exist in the hashif(DateTime->compare($entry->{date},$undefined_date) == 0) {$entry->{day} ="(no date found)";}else{$entry->{day} =$entry->{date}->clone->set_time_zone('UTC')->ymd;# operate on a clone}my@categories=map{ xml_escape strip_html($_->to_literal) }$xpc->findnodes('category | atom:category/@term',$element);$entry->{categories} =@categories? \@categories:undef;# key must exist in the hash$entry->{excerpt} ='';$entry->{content} ='';my@nodes=$xpc->findnodes('description[text()!=""] | atom:content[text()!=""]',$element);@nodes=$xpc->findnodes('summary[text()!=""] | atom:summary[text()!=""] | itunes:summary[text()!=""]',$element)unless@nodes;my$content=shift(@nodes);# The default is that the content is either plain text or escaped HTML,# which is what we want for RSS. For Atom feeds, type="xhtml" means that the# content is XHTML, so it needs to be escaped.if($content) {my$is_xhtml=$content->hasAttribute("type") &&$content->getAttribute("type") eq"xhtml";$entry->{excerpt} = excerpt($content->to_literal);formy$child($content->childNodes()) {my$c=$child->toString();$c= xml_escape$cif$is_xhtml;$entry->{content} .=$c;}}}}subexcerpt {my$content=shift;return'(no excerpt)'unless$content;my$doc=eval{ XML::LibXML->load_html(recover=> 2,string=>$content) };my$separator="¶";formy$node($doc->findnodes('//style | //script | //time | //*[contains(@class, "post-header")] | //*[contains(@class, "post-share-buttons")] | //*[contains(@class, "post-footer")] | //*[contains(@class, "comments")]'),){$node->parentNode->removeChild($node);}formy$node($doc->findnodes('//p | //br | //blockquote | //li | //td | //th | //div | //h1 | //h2 | //h3 | //h4 | //h5 | //h6')) {$node->appendTextNode($separator);}my$text= strip_html($doc->textContent());$text=~ s/( +|----+)/ /g;# collapse whitespace and trim$text=~ s/\s+/ /g;$text= trim$text;# replace paragraph repeats with their surrounding spaces$text=~ s/ ?¶( ?¶)* ?/¶/g;$text=~ s/^¶//;$text=~ s/¶$//;my$len=length($text);$text=substr($text, 0, 500);$text.="…"if$len> 500;$text= xml_escape$text;$text=~ s/¶/<span class="paragraph">¶ <\/span>/g;return$text;}# When there's a value that's supposed to be text but isn't, then we can try to# turn it to HTML and from there to text... This is an ugly hack and I wish it# wasn't necessary.substrip_html {my$str=shift;return''unless$str;my$doc=eval{ XML::LibXML->load_html(string=>$str) };return$strunless$doc;return$doc->textContent();}# Strip RSS 2.0 emailssubwithout_email {my$str=shift;$str=~ s/\S+@\S+ \(([^\)]+)\)/$1/;return$str;}=head1 SEE ALSOOPML 2.0, L<http://dev.opml.org/spec2.html>RSS 2.0, L<https://cyber.harvard.edu/rss/rss.html>Atom Syndication, L<https://tools.ietf.org/html/rfc4287>River of News,=cut1;