CSV Encoding Problems: UTF-8 and Windows-1252, Tested

By Junaid Aslam. 6 min read. Updated .

A CSV file is plain text, but the text has an encoding, and the file rarely says which one. The CSV Cleaner guesses from the bytes. In our test it repaired a small Windows-1252 file on the first try and missed a 10,654-byte one, because the same three bad bytes were a smaller fraction of the file.

This guide shows the five test files, the detected encoding in every run, the exact rows that changed, and the button that fixed the miss. It also shows what the tool exports, because the export decides what the next program sees.

The five test files

FileBytesEncodingWhat it holds
orders-utf8.csv185UTF-8, no BOMFive lines with José Álvarez, Anna Müller, 北京, ☕, and 🙂
orders-utf8-bom.csv188UTF-8 with BOMThe same bytes, with EF BB BF in front
orders-cp1252-small.csv190Windows-1252Four rows of Western names: José, Müller, François, Søren
orders-cp1252-large.csv10,654Windows-1252201 rows, almost all ASCII, with one accented row at the top
orders-cp1252-semicolon.csv151Windows-1252Three rows separated by semicolons, as some European exports are

Each file is a small order table with five columns: order_id, customer, city, amount_eur, and note. We wrote the bytes with a Python script, so each variant is exact: same text, different encoding. The files are not published, and no real customer data is in them.

Why the same letter breaks differently

The letter é is one character with two common byte forms. In UTF-8 it is two bytes, C3 A9. In Windows-1252 it is one byte, E9. A UTF-8 decoder that meets E9 on its own cannot use it, so it emits U+FFFD, the replacement character, which most pages draw as a black diamond with a question mark.

The Unicode FAQ describes that behavior: an ill-formed sequence either signals an error, filters the byte out, or represents the byte with U+FFFD. Browsers take the last option. The text does not become corrupt on disk. It becomes unreadable in that one program, and copying it out of the program can make the damage permanent.

That is why the same file can look fine in one tool and broken in another. The bytes did not change. The decoder did.

How the page decides

The rule is in packages/csv-utils/src/detect.ts. A UTF-8 BOM settles it: the file is UTF-8. Without a BOM, the page decodes the bytes as UTF-8, counts the replacement characters, and switches to Windows-1252 when more than 0.1 percent of the decoded characters are replacements.

One replacement character per 1,000 decoded characters is the cutoff. A file with three bad bytes needs fewer than about 3,000 characters to cross it.

The page then shows the result and offers a retry button. The button decodes the same bytes again with the other encoding, so the original file is not needed a second time.

Results: three runs per file

Every run opened a new browser context, uploaded one file, and read the detected encoding, the detected delimiter, and the first data row. All three runs of a file agreed, so the table shows one row per file and notes the repeats.

FileDetectedDelimiterFirst data rowRuns
orders-utf8.csvUTF-8,1001 José Álvarez Madrid 42.50 café ☕3 of 3
orders-utf8-bom.csvUTF-8,1001 José Álvarez Madrid 42.50 café ☕3 of 3
orders-cp1252-small.csvWindows-1252,1001 José Álvarez Madrid 42.50 café3 of 3
orders-cp1252-large.csvUTF-8,900 Jos� �lvarez Madrid 42.50 caf�3 of 3
orders-cp1252-semicolon.csvWindows-1252;1001 José Álvarez Madrid 42.50 café3 of 3

Four of the five files came out right. The UTF-8 files kept the emoji and the Chinese city name. The small Windows-1252 file was detected and repaired without a click. The semicolon file kept its delimiter and its accents.

The large Windows-1252 file was detected as UTF-8 and showed three replacement characters in the first row. The reason is arithmetic, not luck. It has 10,654 decoded characters and three invalid bytes: 0.028 percent, below the 0.1 percent cutoff. The small file has 190 characters and 13 invalid bytes: 6.8 percent, well above it.

The repair

Clicking Try Windows-1252 instead on the large file changed the detected encoding to Windows-1252 and the row to 900 José Álvarez Madrid 42.50 café, with all three accents correct. The retry worked in the first run, and the two repeat runs confirmed the miss is stable, not intermittent.

The lesson is to look at the encoding label, not only at the table. A table full of replacement characters is a signal. So is a name that looks like José, which is the same failure seen through a different decoder.

What the export does

After cleaning, the page exports the cleaned table. We downloaded the CSV three times, read the bytes, and uploaded the export back into the page.

SourceExport bytesBOMLine endingsRe-import
orders-utf8.csv188NoCRLFUTF-8, accents and emoji intact
orders-cp1252-small.csv206NoCRLFUTF-8, accents intact
orders-cp1252-large.csv, after repair10,857NoCRLFUTF-8, all accents intact

The export is always UTF-8, with CRLF line endings and no byte order mark. RFC 4180 names CRLF as the record separator and registers text/csv, but it also says CSV has no single master specification and that implementations differ. The export follows the common form.

The missing BOM is the part worth knowing. The Unicode FAQ describes EF BB BF as a signature that marks a file as UTF-8, and notes that some programs do not expect it. A reader that assumes an older local code page has nothing to go on. The export is correct UTF-8 either way, and the re-import proved it: the page read its own export as UTF-8 and kept every accent.

What to do with a broken file

  • Upload the file, then read the encoding label before you trust the table.
  • If names show replacement characters, click Try Windows-1252 instead. The bytes are still there, so the retry can recover them.
  • If the retry makes the file worse, click the button again to go back. The page keeps the original bytes in memory.
  • When you save your own CSV for another program, choose UTF-8 with a BOM if that program has shown trouble with accents. The BOM is the signature the Unicode FAQ describes.
  • Keep the original file until the cleaned export looks right in the next program. A re-saved copy can bake the wrong characters in.
We did not open these files in Excel or Notepad. This guide reports the browser, the page, and the bytes only, so it makes no claim about how a spreadsheet application displays them.

Tested on

2026-11-17, against commit 0a0a237 on a local production build (Next.js 14.2.35). Browser: Chromium 159.0.8154.6 through Playwright 1.65.1 on Node 24.20.0. Host: Windows 10 desktop. Each file was uploaded three times in a fresh context, and each export was downloaded once and read with a byte reader.

Primary sources

Document sources and photo credits

This guide has no photos. The five test files were written for this test with a Python script, and they contain invented order rows: no real customer names, addresses, or orders. The file sizes and the detected values in the tables come from those files on the date above.

Related tools

  • CSV Cleaner: upload, repair the encoding, and export CSV, JSON, Excel, or Markdown.
  • JSON Formatter: format or query a JSON file in the browser.
  • Text Diff: compare two versions of a file and see exactly which characters changed.
  • Image Converter: convert a photo without an upload.

Try it yourself

More guides