A lowercase redirect breaks indexing of URLs with non-ASCII characters: a Googlebot test

The idea of the test

“Redirect all URLs to lowercase” is one of the most common pieces of technical SEO advice. It removes duplicates like /Page/ and /page/ and really does work for plain Latin URLs. The test checks what such a rule does to URLs that contain other characters. I used Cyrillic as the example.

A Cyrillic URL reaches the server in percent-encoding. Each UTF-8 byte is written as % and two hex digits, so the letter “т” becomes %D1%82. A rule that lowercases the raw request string lowercases these digits too: %D1%82 turns into %d1%82.

This isn’t limited to Cyrillic. The first byte of any non-ASCII character in UTF-8 has a value from C2 to F4, so its encoding always contains a hex letter. Georgian, Greek or Arabic URLs are affected, as are Latin letters with diacritics (é = %C3%A9, ł = %C5%82) and encoded symbols such as a comma %2C. Only codes made of digits alone stay the same, such as a space %20.

Under RFC 3986 both spellings mean the same thing, but the standard recommends converting hex digits to uppercase. If a crawler does exactly that, after the lowercase redirect it ends up back where it started. The test was meant to show whether Googlebot behaves this way.

How the test was set up

The test object was an ordinary-looking page, /технічний-аудит/ (“technical audit”), with a short technical SEO audit checklist. Its content says nothing about the test, encoding or redirects. I made the page ordinary on purpose, so that only the URL, not the content, would affect crawler behaviour.

A lowercase redirect runs on the server for this URL. If the raw request string differs from its lowercase spelling, the server responds with a 301 to it. For the standard encoding the result is the same as from a rule that takes $request_uri in nginx and lowercases it: %D1%82 becomes %d1%82.

In practice this gives the following responses:

Request	Response
/%D1%82%D0%B5…%D1%82/ (standard encoding, uppercase hex)	301 to lowercase
/%d1%82%d0%b5…%d1%82/ (all lowercase)	200
lowercase with a ?utm_source=… parameter	200

The canonical tag points to the lowercased form, as sites with a lowercase rule usually do. A URL with Latin letters and no diacritics or encoded symbols has no percent-encoding, so such a redirect is safe for it.

How Google was meant to find the URL

Until 25 October there were no links to this page on the site. The URL reached Google in two ways, and both gave it the lowercased form unchanged:

• an inspection in URL Inspection in Search Console, where I entered exactly the lowercased form;
• a separate sitemap-experiments.xml file with the lowercased URL, listed in robots.txt.

These two paths show whether Google changes a URL that it was given already in lowercase.

On 25 October I added a link to the page in this card and in the article about the mechanics. The link uses the standard encoding with uppercase hex, as any site would write it. I’ll add to the results how Googlebot handles the internal link.

Results

The data comes from the nginx log from 17 to 24 October 2026. The table lists every Google request to the page and a few other telling clients:

Date	Client	Requests	What happened
17.10	Chrome	uppercase hex → 301, then lowercase → 200	The page opened after one redirect
19.10	Google-InspectionTool	6 × uppercase hex → 301 within one second	Didn’t request the lowercased form
20.10	URL Inspection with the lowercased URL as input, twice	24 × uppercase hex → 301	Search Console rewrote the URL before requesting it
20.10	Googlebot	sitemap-experiments.xml → 200	Read the lowercased URL
20.10	ClaudeBot	1 × lowercase → 200	Took the URL from the sitemap unchanged
21.10	Googlebot Smartphone	3 × uppercase hex → 301 within two seconds	Didn’t request the lowercased form
21.10	QlyzeBot	lowercase → 200	Took the URL unchanged
23.10	Googlebot Smartphone	3 × uppercase hex → 301	Didn’t request the lowercased form

In total Google requested the page 36 times. All 36 requests went to the uppercase-hex form, and all got a 301. No Google client requested the lowercased form, even though it was both in the Location header and in the sitemap.

The logs show a steady pattern. In URL Inspection each pass through the loop by one agent takes exactly six requests: the first one and five more attempts after the redirect. On 20 October I inspected the URL twice, and each time Google went through the loop separately with a mobile and a desktop agent, which made 24 requests. Regular Googlebot makes exactly three requests per visit and stops.

Example: one Googlebot visit

Here are three lines from the nginx log for 23 October. The path is shortened, the rest is left as is:

23/Oct/2026:04:26:22  192.178.4.3  "GET /%D1%82%D0%B5%D1%85…%D1%82/"  301  "…(compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
23/Oct/2026:04:26:24  192.178.4.4  "GET /%D1%82%D0%B5%D1%85…%D1%82/"  301  "…(compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
23/Oct/2026:04:26:24  192.178.4.3  "GET /%D1%82%D0%B5%D1%85…%D1%82/"  301  "…(compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Every response contained a Location with the lowercase URL. Googlebot’s next request went to uppercase hex again. After the third request it stopped without ever getting the content.

How I checked that it was Google

A User-Agent is easy to fake, so I checked the IP addresses through DNS, as Google recommends. All Google requests came from three addresses: 192.178.4.3, 192.178.4.4 and 192.178.4.5. Reverse DNS for them returns names like crawl-192-178-4-x.googlebot.com, and a forward lookup of the name returns the same IP.

This check filtered out one fake request. On 23 October a “bingbot” read the sitemap from a Google Cloud IP address, while the real Bing uses Microsoft networks. I left that request out of the results.

Conclusions

A lowercase redirect that lowercases the raw request string breaks indexing of pages with non-ASCII characters in the URL. Cyrillic is just the example in this test. Google converts hex digits to uppercase before the first request, whatever source it took the URL from. So the lowercased form is out of its reach, and a redirect to it turns into a loop.

A person won’t notice this problem. In my test, Chrome took the URL from Location as is and opened the page after one redirect. ClaudeBot and QlyzeBot got the page as well, so it turned out broken only for Google.

You don’t have to give up the lowercase redirect, but it’s worth testing on non-ASCII URLs. Here is what to look at:

• the rule lowercases the decoded path, not a string that contains %XX;
• URLs that differ only in hex case don’t get a redirect;
• canonical and the sitemap contain the standard form with uppercase hex;
• the uppercase-hex URL answers 200 or leads to a 200 in one step.

The simplest check takes a minute. Request a URL with Cyrillic, diacritics or an encoded symbol in the standard uppercase-hex encoding using curl -sI and look at the Location header. If lowercase hex appears there, your site has the same problem.

What’s next

I’m still monitoring the test. I’m waiting to see what the Pages report in Search Console shows for this URL and how Googlebot follows the internal link. The real Bingbot hasn’t visited the page yet, so I’ll add its behaviour separately.

Sources

The mechanics rely on the URI standard and Google’s documentation. I took the behaviour of the browser and crawlers from my own server’s logs. Here are the primary sources:

• RFC 3986: section 2.1 on percent-encoding and section 6.2.2.1 on normalizing the case of hex digits;
• Google: Redirects and Google Search;
• Google: Verifying requests from Google’s crawlers;
• Google: Common crawlers, including Google-InspectionTool;
• Search Console Help: URL Inspection tool.

Why this happens, how to set up a lowercase redirect safely and how to check your site is covered in the article “How a lowercase redirect breaks indexing of URLs with Cyrillic and other non-ASCII characters”.