fix: stop emitting the literal string "None" for undecodable text

[hylin (linhongyu510)]

Summary

CsvConverter and PlainTextConverter can emit the literal three-character
string None as document content when charset detection fails, with no error
raised.

Both fall back to detection when no charset is declared:

content = str(from_bytes(file_stream.read()).best())

charset_normalizer.from_bytes(...).best() returns None when it cannot settle
on an encoding, and str(None) == "None".

Details

Reproduced on main (a035350) with bytes that defeat detection —
b"\xff\xfe" + bytes([0xD8, 0x00, 0xDC]):

input	before	after
.csv	| None | | --- |	| \x00 | | --- |
.txt	None	\x00
.md	None	\x00
.json	None	\x00

The failure mode is what makes this worth fixing: the output is not empty and
not an error, it is a well-formed one-cell Markdown table containing a word
that never appeared in the source. Anything consuming the result downstream
(an index, an LLM prompt) sees None as real document text.

This is already handled correctly elsewhere in the repo, so the fix follows
existing precedent rather than inventing a policy:

• _markitdown.py:751 — if charset_result is not None:
• _outlook_msg_converter.py:290 — if detected is not None: … else
  data.decode("utf-8", errors="ignore")

Both converters now use that same lossy-decode fallback. Undecodable bytes
degrade instead of fabricating content, and the declared-charset and
successful-detection paths are untouched.

Scope note: grep -rn 'str(from_bytes(.*).best())' finds exactly these two
call sites, so this covers all of them.

Related Issues

None found — searched open issues for None / charset / undecodable and found
no existing report. Discovered while probing the text converters directly.

How to Validate

cd packages/markitdown
pip install -e ".[all]" pytest
python -m pytest tests/test_undecodable_text.py -q

Expected: 7 passed.

To confirm the tests are load-bearing, stash the two converter changes and
re-run with the new test file in place:

4 failed, 3 passed

The 4 failures are the undecodable assertions. The 3 that still pass are
deliberate guards: one asserts the fixture really is undecodable (so a future
charset_normalizer that starts decoding these bytes cannot quietly turn the
suite green), and two cover the Shift-JIS detection and declared-charset paths
to show the fallback does not swallow the normal cases.

Full suite, before and after this change:

before: 438 passed, 5 failed, 4 skipped
after:  445 passed, 5 failed, 4 skipped

The 5 failures are identical in both runs and unrelated to this change — they
are missing optional dependencies in my local env (azure-ai-contentunderstanding,
azure-ai-documentintelligence, speech_recognition, pydub), which I could
not install because this machine runs Python 3.14 and youtube-transcript-api~=1.0.0
in the all extra requires <3.14. CI runs 3.10–3.13, where [all] installs
cleanly, so those 5 should not appear there.

black reports all three files unchanged.

[hylin (linhongyu510)]

@microsoft-github-policy-service agree

[hylin (linhongyu510)]

Rebased onto the latest main (4459ed0) and force-pushed with lease; head is now 7a99eb7. The branch was BEHIND by one commit that only touched the pptx converters, so there was no overlap with the csv/plain-text converters changed here and the diff is unchanged (+96/−2).

Verified on the rebased head: pytest packages/markitdown/tests/test_undecodable_text.py — 7 passed.

For the wider suite I compared against a clean origin/main worktree in the same environment: both the base and this branch report the same 126 failures, which come from optional converter dependencies missing locally (pdf/docx/lxml extras), not from this change. The branch differs only by the 7 additional passing tests added here. git diff --check is clean.

Ready for review when convenient.