• lud@lemm.ee
    link
    fedilink
    arrow-up
    3
    ·
    5 months ago

    That’s a shame, It would be more interesting to run your own crawlers.

    (Yes I realise that would be computationally intense and hard, but it would nonetheless be cool)

    • whaleross@lemmy.world
      link
      fedilink
      arrow-up
      3
      ·
      5 months ago

      There is probably a neat wget oneliner that could crawl everything on the open web. The real challenge is how to index all the information. That might be a neat Perl oneliner.