Heritrix Content Filtering

Question

I have a requirement to aggregate content from several different web sites (primarily HTML pages and PDF documents). I'm currently experimenting with Heritrix (3.2.0) to see if it will meet my needs.

While the documentation is pretty detailed the engine does not seem to be working as i would expect. I have setup some simple jobs and configured DecideRules many different ways, but no matter what I do I find that Heritrix is either pulling down way too much content or nothing at all.

Here's an example of what i'm trying to do. I'm pointing Heritrix to URL like this...example.com/news/speeches. This is a web page that has an HTML table with links to individual speeches (ex..example.com/news/speech/speech1.html, xample.com/news/speech/speech2.html, etc, etc). I really only need the HTML and PDF docs one level down from the parent page. I want to prevent Heritrix from navigating deeper than 1 level, prevent it from pulling content if not below this specific path on the example.com domain, preventing it from navigating out to another domain, and restrict it to html and pdf content.

The following configuration is what I would think should work but doesn't

 <bean id="longerOverrides" class="org.springframework.beans.factory.config.PropertyOverrideConfigurer">
      <property name="properties">
       <props>
        <prop key="seeds.textSource.value">

    # URLS HERE
    example.com/news/speeches

        </prop>
       </props>
      </property>
     </bean>

<bean id="scope" class="org.archive.modules.deciderules.DecideRuleSequence">
  <!-- <property name="logToFile" value="false" /> -->
  <property name="rules">
   <list>
    <!-- Begin by REJECTing all... -->
    <bean class="org.archive.modules.deciderules.RejectDecideRule">
    </bean>
    <!-- ...then ACCEPT those within configured/seed-implied SURT prefixes... -->
    <bean class="org.archive.modules.deciderules.surt.SurtPrefixedDecideRule">
     <!-- <property name="seedsAsSurtPrefixes" value="true" /> -->
     <!-- <property name="alsoCheckVia" value="false" /> -->
     <!-- <property name="surtsSourceFile" value="" /> -->
     <!-- <property name="surtsDumpFile" value="${launchId}/surts.dump" /> -->
      <property name="surtsSource">
           <bean class="org.archive.spring.ConfigString">
            <property name="value">
             <value>
            example.com/news/speeches
             </value>
            </property> 
           </bean>
          </property>
    </bean>
     <!-- ...and REJECT those from a configurable (initially empty) set of URI regexes... -->
<bean class="org.archive.modules.deciderules.MatchesListRegexDecideRule">
      <property name="decision" value="REJECT"/>
      <property name="listLogicalOr" value="true" />
      <property name="regexList">
       <list>
         <value>.*(?i)(\.(avi|wmv|mpe?g|mp3))$</value>
         <value>.*(?i)(\.(rar|zip|tar|gz))$</value>
         <value>.*(?i)(\.(xls|odt))$</value>
         <value>.*(?i)(\.(xml))$</value>
         <value>.*(?i)(\.(txt|conf|pdf))$</value>
         <value>.*(?i)(\.(swf))$</value>
         <value>.*(?i)(\.(js|css))$</value>
         <value>.*(?i)(\.(bmp|gif|jpe?g|png|svg|tiff?))$</value>
       </list>
      </property>
</bean>
    <!-- ...but REJECT those more than a configured link-hop-count from start... -->
    <bean class="org.archive.modules.deciderules.TooManyHopsDecideRule">
     <!-- <property name="maxHops" value="20" /> -->
    </bean>
    <!-- ...but ACCEPT those more than a configured link-hop-count from start... -->
    <!--bean class="org.archive.modules.deciderules.TransclusionDecideRule"-->
     <!-- <property name="maxTransHops" value="2" /> -->
     <!-- <property name="maxSpeculativeHops" value="1" /> -->
    <!--/bean-->
    <!-- ...but REJECT those from a configurable (initially empty) set of REJECT SURTs... -->
    <bean class="org.archive.modules.deciderules.surt.SurtPrefixedDecideRule">
          <property name="decision" value="REJECT"/>
          <property name="seedsAsSurtPrefixes" value="false"/>
          <property name="surtsDumpFile" value="${launchId}/negative-surts.dump" /> 
     <!-- <property name="surtsSource">
           <bean class="org.archive.spring.ConfigFile">
            <property name="path" value="negative-surts.txt" />
           </bean>
          </property> -->
    </bean>
    <!-- ...and REJECT those from a configurable (initially empty) set of URI regexes... -->
    <bean class="org.archive.modules.deciderules.MatchesListRegexDecideRule">
          <property name="decision" value="REJECT"/>
     <!-- <property name="listLogicalOr" value="true" /> -->
     <!-- <property name="regexList">
           <list>
           </list>
          </property> -->
    </bean>
    <!-- ...and REJECT those with suspicious repeating path-segments... -->
    <bean class="org.archive.modules.deciderules.PathologicalPathDecideRule">
     <!-- <property name="maxRepetitions" value="2" /> -->
    </bean>
    <!-- ...and REJECT those with more than threshold number of path-segments... -->
    <bean class="org.archive.modules.deciderules.TooManyPathSegmentsDecideRule">
     <!-- <property name="maxPathDepth" value="20" /> -->
    </bean>
    <!-- ...but always ACCEPT those marked as prerequisitee for another URI... -->
    <bean class="org.archive.modules.deciderules.PrerequisiteAcceptDecideRule">
    </bean>
    <!-- ...but always REJECT those with unsupported URI schemes -->
    <bean class="org.archive.modules.deciderules.SchemeNotInSetDecideRule">
    </bean>
   </list>
  </property>
 </bean>

I expected my crawl to only pull down a dozen or so html documents as that is all that's contained within the /speech path. After about a half hour i stopped my crawl as it was downloading 800+ documents, as I found it was traversing backwards up to parent level paths. I also experimented with RegEx rules with no luck. Any help would be appreciated.

Answer 1

One good thing to debug such issue, is to enable the logging of the scope decisions. (Uncomment line with logToFile and make it true. This will provide your for every URI the rule that made the decision to include or to reject it. Thus you will be able to see which of your rule is not properly configured and accepts URI that should have been rejected.