Internet activity

[Marauder2003]

GoDaddy says my website has 7% robot devices accessing it. Don?t know if that activity includes robots that pay attention to my robots.txt file.

This makes me wonder. How much of the overall internet activity is robots, spiders, etc? And now with, I assume, AI looking for stuff to learn.

[musick]

The CPO of the company I am working for recently gave a presentation on this very topic. He showed that bot traffic makes up about 44% of all internet activity in 2022 and bad bot traffic is double that of good bots that perform legitimate functions such as indexing and automated responses. But it also depends on the popularity and content of the website or the specific account on the website. With more views or popularity comes increased activity.

[OlderThanDirt]

Besides blocking entire countries, I have hundreds of subnets blocked. One suspicious IP address bans the entire subnet.

[Marauder2003]

GoDaddy allows me to block by country. But somehow China is still my number 2 hitter.

Besides blocking entire countries, I have hundreds of subnets blocked. One suspicious IP address bans the entire subnet.

[uzigalil]

Go daddy is pure garbage

[The Hamsterball]

You can set up a Google Analytics 4 property and it will filter out bot activity.

There is a specific county (that would show up in the location of the filter for the region the hit or traffic is coming from).
This county is made up of a bunch of servers containing bots that go to every website or websites.

I forget the county. But I get hits from the county literally every day.

In your google analytics property you can set a filter to exclude these and then segment out what your true traffic is. There is no need to find the IP's. Just the location.

I would have to check my Google Analytics connections and find the exact county, state for you, but it is not that hard to find.

[The Hamsterball]

You can set up a Google Analytics 4 property and it will filter out bot activity.

There is a specific county (that would show up in the location of the filter for the region the hit or traffic is coming from).
This county is made up of a bunch of servers containing bots that go to every website or websites.

I forget the county. But I get hits from the county literally every day.

In your google analytics property you can set a filter to exclude these and then segment out what your true traffic is. There is no need to find the IP's. Just the location.

I would have to check my Google Analytics connections and find the exact county, state for you, but it is not that hard to find.

Okay, so the city is "Ashburn, Virginia".
If you filter out hits from this city/county, you will remove most of the BOTS from your web activity results.

For others, you will have to figure out the other attributes and filter those ones out accordingly too.

[Marauder2003]

I think what you are talking about is massaging the log data. I want to stop the activity from getting to my shared server. I don?t have access to the firewall in that way needed to block activity. Since my log shows activity from China I can only assume the setting I do have access to does not block as well as I would like.

Okay, so the city is "Ashburn, Virginia".
If you filter out hits from this city/county, you will remove most of the BOTS from your web activity results.

For others, you will have to figure out the other attributes and filter those ones out accordingly too.

[The Hamsterball]

I think what you are talking about is massaging the log data. I want to stop the activity from getting to my shared server. I don?t have access to the firewall in that way needed to block activity. Since my log shows activity from China I can only assume the setting I do have access to does not block as well as I would like.

The only way you can block the data (if it is indeed BOTS from other countries) is by setting a robot/identity verifier at the beginning of entrance to your website homepage.

But this would make every single person have to confirm they are a real person before accessing your website.

I can understand if you're paying for certain loads of traffic, however I myself pay a flat rate per year for all traffic no matter how much or how little I get, plus the cost of my e-mail service.

I think the main concern I would worry about is if you get mass loads of traffic that slow down your web domain, which would happen in something like a DDOS attack.

However, if your page is loading fine and not too slow and your server's resources are not overly taxed, the BOTS are not something to worry about.

I have been at places when such activity happened and it's really scary, and everything shuts down and the infrastructure team has to scramble to block the activity.

If that is not happening, I would not worry about it, because there are BOTS all over the world accessing the entire web.

[mshill]

The only way you can block the data (if it is indeed BOTS from other countries) is by setting a robot/identity verifier at the beginning of entrance to your website homepage.

But this would make every single person have to confirm they are a real person before accessing your website.

I can understand if you're paying for certain loads of traffic, however I myself pay a flat rate per year for all traffic no matter how much or how little I get, plus the cost of my e-mail service.

I think the main concern I would worry about is if you get mass loads of traffic that slow down your web domain, which would happen in something like a DDOS attack.

However, if your page is loading fine and not too slow and your server's resources are not overly taxed, the BOTS are not something to worry about.

I have been at places when such activity happened and it's really scary, and everything shuts down and the infrastructure team has to scramble to block the activity.

If that is not happening, I would not worry about it, because there are BOTS all over the world accessing the entire web.

Why wait until it happens? It's like riding a motorcycle, it's not if you go down but when.

[The Hamsterball]

Why wait until it happens? It's like riding a motorcycle, it's not if you go down but when.

They have to have something to gain from hacking your website or flooding it with traffic rendering it unusable.

That depends on the size of your business. The purpose. Who the likely organizations or hackers are part of.

My businesses are mainly for entertainment and doing things in my free time and getting a little money to pay for expenses to run them. On the side.

I also don't have a lot of capital. They are really tiny and there's no money involved except for certain services I provide.


Now if you are the web developer for a FFL or big name chain.. there could be organizations against gun lobbying or that want the dealers to shut down, possibly of the variant of people that are against 2nd Amendment. Who not only have those qualities, but also have hackers in their network.

But they would have to be the type of hackers that have resources to expend putting effort into taking down your infrastructure. This may happen to a few chain FFL's I know of whose websites regularly go down.

However, the name of the game in protection is knowing who would come after you. They have to benefit in some way .

There is a cost vs benefit analysis as well. If you know your users are a-ok with verifying this, verifying that, each time they want access to the contents of your page, then sure.

It all depends on what you have out there.