Workable Bulletin Board / RSS streaming reader...?

[RCxRC]

I am annoyed that after literally months of trying a variety of supposed hi-quality RSS streaming reader / aggregator apps (such as for reading the CG VBulletin RSS feed) that I still have not found a solution to being notified of specific search terms on the CG website forums in a reasonably short time period (typically less than 10-45 min). Some of these have been $50-100 / year upper-end RSS reader services, and after trying them out and inquiring of their tech-support, it appears nothing will work exactly as I am trying to get it to.

The issue: trying to find an "app" or RSS streaming newsfeed reader that will search for specific search terms (ie, "Freedom Arms", "K98", etc) on specific sub-boards on CG forums and notify me in a timely manner (via text, email, other notification, etc).

I am tired of missing out on nearly everything that gets posted on the Marketplace sub-forums (long guns, handguns). I can't sit by the computer and do a search every 5-10 minutes for specific search terms for the half-dozen things I am looking for to round-out my collection. Nor do I want to have to keep refreshing my screen every couple of minutes to see what is new. I do have to work.

I have looked high and low for a solution and have not found one that works.

I tried also tried Google search / saved search. That doesn't notify in a timely manner.

Of all the RSS feed readers that I have tried to harness CG's RSS feed to, none is adequate. Feed Rinse, VisualPing (which wants close to $60/mo to conduct feed searches at a 5 min interval frequency https://visualping.io/pricing ), IFTTT, Feedly Pro, RSS Owl, etc. The closest I got was to get some (not all, though) posts from the sub-forums to post to the app every couple of hours (and subsequently alert me), and never sooner than 10 min up to a half-hour at best. I am starting to believe the issue is with the CG RSS feed, in that it does not send updated additions to the website in a timely manner (at least not fast enough to be useful for this purpose). Another issue is that even the fastest setting on most of the various RSS reader services only goes down to 10-15 minutes (with the exception of VisualPing, at 5 min, but at a steep premium, and RSS Owl, at 1 min intervals). Of all the ones I've tried, RSS Owl (now not recommended due to security issues) allowed the most frequent checks (as low as once a minute), but results would only post through the RSS feed from CG to the app in between 10-30 minutes (typically). I noticed the same thing from Feedly Pro. I would actually wait for a new post to show up on CG, then watch and wait until it showed up in the app. And that's with checks down to once a minute. I am starting to think the RSS route is useless for my purpose b/c the feed isn't updating constantly or even every minute or less, but instead in between 10-30 minutes on average (8 min at the very least, and up to 45 min at the very latest, from dozens of posts I compared).

Has anyone found a solution that works for this purpose? Or is this just a closely guarded secret for the swoop-in / "I'll take it" now crowd that always seems to get on everything that posts minutes after it goes live....?

The last time I deep-dived into this several months ago, I found a few threads on a techie website where they were hacking either VBulletin or a RSS reader service with a VB plug-in script that sped things up and made notifications more specific and efficient. I had also found a post referring to a "website scubber / scraper" that is apparently a popular first programming project while learning Python. (Note to self: A Google search of "Python RSS Feed Scraping" or "Python Website Scraping" leads to a ton of online tutorials...will be investigating...)

I know there has to be a way.....

Apparently this has been brought up a few times over the past few years, but none of these solutions have worked for me: https://calguns.net/calgunforum/showthread.php?t=974490 , http://www.calguns.net/calgunforum/s...d.php?t=842185 , https://www.calguns.net/calgunforum/...ad.php?t=25692 , https://calguns.net/calgunforum/showthread.php?t=214896 .

In the interest of full disclosure, for anyone interested in trying the same or an alternative solution (perhaps notifications every 10-30 min on average is fine for your needs), here are the RSS Feed url's for the Marketplace / Long Gun and Marketplace / Handgun forums:

CalGuns Marketplace Long Guns

http://www.calguns.net/calgunforum/external.php?type=rss2&forumids=92

CalGuns Marketplace Hand Guns

http://www.calguns.net/calgunforum/external.php?type=rss2&forumids=332

FOLLOW-UP NOTE: Well, I got part of the answer. Apparently RSS feed rates are set by the vBulletin site administrator....typical rate apparently defaults to once an hour, and also defaults to only pushing the very first post of a new thread (no f/u posts, unless specified) https://forum.vbulletin.com/forum/vb...pdating-slowly . Sooooo, off we go to discover a website scraper program option.....into the abyss....

Addl note: I am discovering references to "aggressive website scraping" (such as scraping an entire website for data) which could result on one being banned from a site https://github.com/IanLondon/detectorist-scraper . I didn't know this was a thing, but as I read more, I can see why constantly "searching" a site could greatly increase its bandwidth usage, raising some issues. It seems a "balance" must be met. One website scraping app author posted his app on Github and noted in the release notes that he even checked in advance with the specific website's forum administrator for an "allowable limit" (permission) on how frequently he could "scrape" the administrator's site ("x" seconds between searches) https://github.com/vizzerdrix55/web-...Bulletin-forum . Perhaps 10-30 min delayed notifications (off the existing RSS Feed stream update rate) aren't too out-of-line, at the risk of being banned. Fair warning!

Addl note: Geeezzz. Didn't know I don't even need to learn how to program this. I can just hire someone! Cheap, too! https://www.freelancer.com/projects/...sw-bypass=&w=f

Link to FAQ info on CG RSS feeds: https://www.calguns.net/calgunforum/...l&titlesonly=0

On a very different side note, I do see a great value in having a "clean" RSS Feed reader to only show "new" threads (newly posted items), instead of the 80-90% of threads in the Marketplace forums which are re-posts. Other than the situation where one puts up multiple pages in their ad for multiple items, adds another post with photos, lowers the price, etc, it really helps clean up a quick search of what's new. I looked at the first page of the Marketplace / Handguns ads this morning, and all but 2 new ads were re-posts, and it was just those two that made their way into my RSS Feed reader app (RSS Owl or Feedly). It does make quicker work of searching for "newly listed" items, vs re-posted.

[SkyHawk]

I wrote my own Wget script. It checks every 90 seconds. Sends me email and text alerts on a list of keywords. It keeps track of thread numbers it hits on, and never alerts more than once on any thread. It only checks thread titles on the first page of the subforums I want, so there is no aggressive scraping or bandwidth usage since does not spider every thread or all pages of thread titles.

Works awesome for years now, sniped pretty much everything I wanted from the marketplace with it. These days it just watches the 'Deals' subforum to let me know if anyone posts new deals from a couple of vendors I am interested in.

Runs in DOS batch, non-stop loop

I have variations of this script to watch Amazon for price changes on products I am interested in, because they can be very transient and short lived and no service I have found will alert you with the resolution needed to catch such short-lived price reductions.

I also use it to watch ammo vendors for any change in stock status on ammo I want. It has found me all the ammo I need.

[RCxRC]

That's basically what I am trying to do. I wouldn't classify it as "sniping" per se in the traditional sense (such as AuctionSniper for eBay, which I have used a few times over the years), but simply getting that actual rare, "good / fair" deal that someone posts where they're just trying to get a fair price quickly for something specific but uncommon that I'm actually looking for. Particularly these days, when "fair" isn't even in the lexicon due to all of the external market forces, and where everyone seems to be hustling pretty hard. I'm not desperate enough for the "high-ball" prices; not yet. Fortunately, my collection is pretty much where I want it (as in the safe is getting quite full), and I'm just looking for a few specific models to round things out. I'm just tired of wading (and waiting) through the 99.9% of posts in the Marketplace that don't apply to what I'm looking for. Particulary the average ~95% of ads that are being re-posted.

Fortunately I am all good on ammo and was prepared a long time ago. "Prime" membership and setting up text alerts for "back in stock" items at Targetsportsusa covers any "modern" ammo issues that crop up. COE/FFL 03 also covers a few of the surplus ammo vendors, thankfully. Amazon, check (camelcamelcamel.com works "adequately" for me as far as price drop alerts, but as you said the very best drops are hard to nail down, sometimes just lasting for a half-hour or less).

After you mentioned it I checked out Wget, as well as Curl, Python, etc. All have their pro's and con's. I'm not in IT and haven't done much command line stuff since High School and early college many, many years ago (Unix, Cobol, Fortran, DOS back in the 80's), other than disk imaging command switches here and there. So this will be a new-ish experience.

I also found some info on additional off-the-shelf options here: https://www.quora.com/What-is-the-be...00+-pages-deep