False positive mixed_case_recommended_field for numeric values

[MKuranowski]

Describe the bug

The mixed_case_recommended_field warning is always triggered when a field contains digits, like route_short_name=A1 or trip_short_name=101.

Characters which are not letters should not be considered at all when comparing case.

Steps/Code to Reproduce

Validate a file which uses numbers in fields like route_short_name or trip_short_name.

Expected Results

No mixed_case_recommended_field warning for those fileds.

Actual Results

{
      "code": "mixed_case_recommended_field",
      "severity": "WARNING",
      "totalNotices": 360,
      "sampleNotices": [
        {
          "filename": "routes.txt",
          "fieldName": "route_short_name",
          "fieldValue": "A1",
          "csvRowNumber": 2
        },
        {
          "filename": "routes.txt",
          "fieldName": "route_short_name",
          "fieldValue": "ZA12",
          "csvRowNumber": 3
        },
        {
          "filename": "trips.txt",
          "fieldName": "trip_short_name",
          "fieldValue": "301",
          "csvRowNumber": 2
        },
        {
          "filename": "trips.txt",
          "fieldName": "trip_short_name",
          "fieldValue": "5301",
          "csvRowNumber": 3
        },
        {
          "filename": "trips.txt",
          "fieldName": "trip_short_name",
          "fieldValue": "103",
          "csvRowNumber": 4
        },
        ...
}

Screenshots

No response

Files used

https://mkuran.pl/gtfs/wkd.zip

Validator version

4.1.0

Operating system

Debian 12

Java version

openjdk version "17.0.6" 2023-01-17

Additional notes

No response

[welcome]

Thanks for opening your first issue in this project! If you haven't already, you can join our slack and join the #gtfs-validators channel to meet our awesome community. Come say hi 👋!

Welcome to the community and thank you for your engagement in open source! 🎉

[briandonahue]

@isabelle-dr How do we want to approach this? It looks like in some cases when numbers are present, we don't want to throw the warning:
A1
103B
1205

In those cases we could skip the mixed case validation altogether, but how about longer strings? Should we have a length minimum or number of letters in a string before mixed case is considered? Thinking about strings like:

MY ROUTE 100
route 1 boulevard

etc

[isabelle-dr]

@briandonahue thanks for having a look at this.
After giving this some thought, here is a proposal:
We could modify the logic so we only run this check if the field value contains at least one " " character and if it doesn't contain only numbers.
This means that the following values would be valid: ZA12, 1205, 1205 1.

cc @tzujenchanmbd

[briandonahue]

@isabelle-dr I had a couple thoughts. 1) I think the current implementation has another flaw - I believe it only checks for the 26 letters of the basic alphabet /[a-zA-Z]/ which will not match characters such as àâçéèêëîïôûùüÿñæœ (and probably others).

2. Given the above, I think maybe a better check would be:
   • the value contains more than one "letter" (defined as non-numeric, and non-white-space characters)
   • AND the original value is the same as the value after being converted to all upper OR all lower case (this will of course not affect numbers and white space, but should handle other characters more efficiently than a regex)

We may still get some false positives if there are values such as AA-103 or B-34-C or D-E etc. But it is a recommendation/warning not an error, and it shoudl cut down significantly on false positives for numeric values or single-letter + numbers.

// if more than one non-numeric, non-whitespace character
if((value.length() - countNumericChars(value) - countWhiteSpaceChars(value)) > 1) &&
// AND the value is all upper OR lower case characters
(value.toLowerCase() == value || value.toUpperCase() == value){
    notices.add(new MixedFieldNotice());
}

[isabelle-dr]

I like this idea!
We would still get the notice for something like ZA12, right?
Should we also add the following condition so that we differentiate acronyms (such as ZA12) from the natural text (such as Route 1B)?

• the field value contains at least one " " character

[briandonahue]

I like this idea! We would still get the notice for something like ZA12, right? Should we also add the following condition so that we differentiate acronyms (such as ZA12) from the natural text (such as Route 1B)?

• the field value contains at least one " " character

@isabelle-dr I thought about this, but I think we want to keep it for long words with no spaces such as Lancaster?

Another approach would be to validate any values with strings of more than 1 letter, regardless of spaces. In this case A 101 and 342B would be ignored but RTE 101, RTE23 and route 1B would be flagged. If we don't want something like RTE23 to be flagged, we could be more strict and ignore strings of mixed letters and numbers (but no spaces). So A12387abi would be ignored but ROUTE A2348B would be flagged.

[briandonahue]

After further thought, I think the simplest logic that will cover most cases is to validate any value that has more than one consecutive letter, regardless of numbers or spaces. It will flag things like 123ABC but that seems like it would be uncommon, and it's only a warning... we could also re-visit if users complain, and add some more complex logic, to ignore mixed number/letter strings or something.

Pseudo-code for simpler logic:

// any consecutive non-numeric, non-space characters (2 or more sequential letters)
if (/[0-9 ]{2,}/.test(value)  &&
// AND the value is all upper OR lower case characters
(value.toLowerCase() == value || value.toUpperCase() == value)){
    notices.add(new MixedFieldNotice());
}

[isabelle-dr]

These are very good points, thank you for giving it some thought @briandonahue!
What you shared in the last message sounds great 👍

[briandonahue]

@isabelle-dr I've got this working (modified to allow non-English characters, etc) but I noticed the original issue lists ZA12 as a potential false positive. Thoughts? Should we leave it as a warning that can be ignored, or should we consider some larger allowable number of characters mixed with numbers in a single string 😅