Add a list of false positives to reduce unjustified filtering #2

Closed
opened 2018-11-13 14:08:25 -05:00 by marc-x-andre · 3 comments
marc-x-andre commented 2018-11-13 14:08:25 -05:00 (Migrated from github.com)

When a word contains a profanity, like assassins, the word could be the check inside a dictionary.

When a word contains a profanity, like `assassins`, the word could be the check inside a dictionary.
TwiN commented 2018-11-13 16:18:07 -05:00 (Migrated from github.com)

Yes, this flaw is mentioned in the README:

The downside is that words like assassin, which contain ass, would also be filtered as profane. So in the future, a list of false positives would have to be added.

However, I wouldn't go as far as to have a dictionary of words, as that would significantly impact the performance.

Yes, this flaw is mentioned in the `README`: > The downside is that words like `assassin`, which contain `ass`, would also be filtered as profane. So in the future, a list of false positives would have to be added. However, I wouldn't go as far as to have a _dictionary_ of words, as that would significantly impact the performance.
marc-x-andre commented 2018-11-13 16:25:32 -05:00 (Migrated from github.com)

I don't think it will significantly impact the performance.
With a good implementation, like {a cool buzzword}, it could be super efficient!

I don't think it will `significantly impact the performance`. With a good implementation, like {a cool buzzword}, it could be super efficient!
TwiN commented 2018-11-13 16:53:50 -05:00 (Migrated from github.com)

The Oxford English Dictionary has over 200k words listed, and assuming we add support for multiple languages, it'd definitely have a huge impact on the performance.

I don't think adding a whole dictionary is acceptable performance-wise, but I do think that adding a list of all words that have a profanity in them but aren't necessary profane (e.g. assassins, as you mentioned) would be a great idea, and much more viable.

The Oxford English Dictionary has over 200k words listed, and assuming we add support for multiple languages, it'd definitely have a huge impact on the performance. I don't think adding a whole dictionary is acceptable performance-wise, but I do think that adding a list of all words that have a profanity in them but aren't necessary profane (e.g. `assassins`, as you mentioned) would be a great idea, and much more viable.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
TwiN/go-away#2
No description provided.