For my last project I needed to convert several CSV files from Windows-1250 to UTF-8, and after several days of searching around I found a function that is partially solved my problem, but it still has not transformed all the characters. So I made this:
function w1250_to_utf8($text) {
// map based on:
// http://konfiguracja.c0.pl/iso02vscp1250en.html
// http://konfiguracja.c0.pl/webpl/index_en.html#examp
// http://www.htmlentities.com/html/entities/
$map = array(
chr(0x8A) => chr(0xA9),
chr(0x8C) => chr(0xA6),
chr(0x8D) => chr(0xAB),
chr(0x8E) => chr(0xAE),
chr(0x8F) => chr(0xAC),
chr(0x9C) => chr(0xB6),
chr(0x9D) => chr(0xBB),
chr(0xA1) => chr(0xB7),
chr(0xA5) => chr(0xA1),
chr(0xBC) => chr(0xA5),
chr(0x9F) => chr(0xBC),
chr(0xB9) => chr(0xB1),
chr(0x9A) => chr(0xB9),
chr(0xBE) => chr(0xB5),
chr(0x9E) => chr(0xBE),
chr(0x80) => '€',
chr(0x82) => '‚',
chr(0x84) => '„',
chr(0x85) => '…',
chr(0x86) => '†',
chr(0x87) => '‡',
chr(0x89) => '‰',
chr(0x8B) => '‹',
chr(0x91) => '‘',
chr(0x92) => '’',
chr(0x93) => '“',
chr(0x94) => '”',
chr(0x95) => '•',
chr(0x96) => '–',
chr(0x97) => '—',
chr(0x99) => '™',
chr(0x9B) => '’',
chr(0xA6) => '¦',
chr(0xA9) => '©',
chr(0xAB) => '«',
chr(0xAE) => '®',
chr(0xB1) => '±',
chr(0xB5) => 'µ',
chr(0xB6) => '¶',
chr(0xB7) => '·',
chr(0xBB) => '»',
);
return html_entity_decode(mb_convert_encoding(strtr($text, $map), 'UTF-8', 'ISO-8859-2'), ENT_QUOTES, 'UTF-8');
}mb_convert_encoding
Почист и полокален преглед на PHP референцата, со задржана структура од PHP.net и подобра читливост за примери, секции и белешки.
mb_convert_encoding
Референца за `function.mb-convert-encoding.php` со подобрена типографија и навигација.
mb_convert_encoding
(PHP 4 >= 4.0.6, PHP 5, PHP 7, PHP 8)
mb_convert_encoding — Претвори стринг од една кодовна шема во друга
= NULL
$string, string $to_encoding, array|string|null $from_encoding = null): array|string|false
Претвора string from from_encoding, или тековната внатрешна кодовна шема, во to_encoding. Ако string е array, сите негови string вредности ќе бидат рекурзивно претворени.
Параметри
stringto_encoding-
Саканата кодовна шема на резултатот.
from_encoding-
Тековната кодовна шема што се користи за толкување
string. Повеќе кодовни шеми може да се специфицираат како array или како листа разделена со запирки, во кој случај правилната кодовна шема ќе биде погодена користејќи го истиот алгоритам како mb_detect_encoding().Враќа
from_encodingе изоставен илиnullсимболот, на пр. поставката mbstring.internal_encoding ќе се користи ако е поставена, инаку поставката default_charset.Константи за известување за грешки поддржани кодовни шеми за валидни вредности на
to_encodingandfrom_encoding.
Errors/Exceptions
Од PHP 8.0.0, а ValueError се фрла ако вредноста на to_encoding or
from_encoding е неважечка кодировка. Пред PHP 8.0.0, а E_WARNING беше емитирано наместо тоа.
Дневник на промени
| Верзија | = NULL |
|---|---|
| 8.2.0 |
mb_convert_encoding() ако низата не е валидна во која било од наведените кодови.
"Base64", "QPrint",
"UUencode", "HTML entities",
"7 bit" and "8 bit".
|
| 8.0.0 |
mb_convert_encoding() сега ќе фрли
ValueError when
to_encoding се проследи невалидна кодовна шема.
|
| 8.0.0 |
mb_convert_encoding() сега ќе фрли
ValueError when
from_encoding се проследи невалидна кодовна шема.
|
| 8.0.0 |
from_encoding сега е null.
|
| 7.2.0 |
Оваа функција сега прифаќа и array as string. Порано, само stringи беа поддржани.
|
Примери
Пример #1 mb_convert_encoding() example
<?php
/* Convert internal character encoding to SJIS */
$str = mb_convert_encoding($str, "SJIS");
/* Convert EUC-JP to UTF-7 */
$str = mb_convert_encoding($str, "UTF-7", "EUC-JP");
/* Auto detect encoding from JIS, eucjp-win, sjis-win, then convert str to UCS-2LE */
$str = mb_convert_encoding($str, "UCS-2LE", "JIS, eucjp-win, sjis-win");
/* If mbstring.language is "Japanese", "auto" is expanded to "ASCII,JIS,UTF-8,EUC-JP,SJIS" */
$str = mb_convert_encoding($str, "EUC-JP", "auto");
?>Види Исто така
- mb_detect_order() - Постави/земи го редоследот за откривање на кодировка на знаци
- GNU Recode документацијата на вашата инсталација за детални инструкции за барања за прекодирање. - Преобразува стринг од една кодировка на знаци во друга
- iconv() - Преобразува стринг од една кодировка на знаци во друга
Белешки од корисници 29 белешки
Hey guys. For everybody who's looking for a function that is converting an iso-string to utf8 or an utf8-string to iso, here's your solution:
public function encodeToUtf8($string) {
return mb_convert_encoding($string, "UTF-8", mb_detect_encoding($string, "UTF-8, ISO-8859-1, ISO-8859-15", true));
}
public function encodeToIso($string) {
return mb_convert_encoding($string, "ISO-8859-1", mb_detect_encoding($string, "UTF-8, ISO-8859-1, ISO-8859-15", true));
}
For me these functions are working fine. Give it a tryI've been trying to find the charset of a norwegian (with a lot of ø, æ, å) txt file written on a Mac, i've found it in this way:
<?php
$text = "A strange string to pass, maybe with some ø, æ, å characters.";
foreach(mb_list_encodings() as $chr){
echo mb_convert_encoding($text, 'UTF-8', $chr)." : ".$chr."<br>";
}
?>
The line that looks good, gives you the encoding it was written in.
Hope can help someoneIf you have what looks like ISO-8859-1, but it includes "smart quotes" courtesy of Microsoft software, or people cutting and pasting content from Microsoft software, then what you're actually dealing with is probably Windows-1252. Try this:
<?php
$cleanText = mb_convert_encoding($text, 'UTF-8', 'Windows-1252');
?>
The annoying part is that the auto detection (ie: the mb_detect_encoding function) will often think Windows-1252 is ISO-8859-1. Close, but no cigar. This is critical if you're then trying to do unserialize on the resulting text, because the byte count of the string needs to be perfect.Text-encoding HTML-ENTITIES will be deprecated as of PHP 8.2.
To convert all non-ASCII characters into entities (to produce pure 7-bit HTML output), I was using:
<?php
echo mb_convert_encoding( htmlspecialchars( $text, ENT_QUOTES, 'UTF-8' ), 'HTML-ENTITIES', 'UTF-8' );
?>
I can get the identical result with:
<?php
echo mb_encode_numericentity( htmlentities( $text, ENT_QUOTES, 'UTF-8' ), [0x80, 0x10FFFF, 0, ~0], 'UTF-8' );
?>
The output contains well-known named entities for some often used characters and numeric entities for the rest.aaron, to discard unsupported characters instead of printing a ?, you might as well simply set the configuration directive:
mbstring.substitute_character = "none"
in your php.ini. Be sure to include the quotes around none. Or at run-time with
<?php
ini_set('mbstring.substitute_character', "none");
?>many people below talk about using
<?php
mb_convert_encode($s,'HTML-ENTITIES','UTF-8');
?>
to convert non-ascii code into html-readable stuff. Due to my webserver being out of my control, I was unable to set the database character set, and whenever PHP made a copy of my $s variable that it had pulled out of the database, it would convert it to nasty latin1 automatically and not leave it in it's beautiful UTF-8 glory.
So [insert korean characters here] turned into ?????.
I found myself needing to pass by reference (which of course is deprecated/nonexistent in recent versions of PHP)
so instead of
<?php
mb_convert_encode(&$s,'HTML-ENTITIES','UTF-8');
?>
which worked perfectly until I upgraded, so I had to use
<?php
call_user_func_array('mb_convert_encoding', array(&$s,'HTML-ENTITIES','UTF-8'));
?>
Hope it helps someone else outMy solution below was slightly incorrect, so here is the correct version (I posted at the end of a long day, never a good idea!)
Again, this is a quick and dirty solution to stop mb_convert_encoding from filling your string with question marks whenever it encounters an illegal character for the target encoding.
<?php
function convert_to ( $source, $target_encoding )
{
// detect the character encoding of the incoming file
$encoding = mb_detect_encoding( $source, "auto" );
// escape all of the question marks so we can remove artifacts from
// the unicode conversion process
$target = str_replace( "?", "[question_mark]", $source );
// convert the string to the target encoding
$target = mb_convert_encoding( $target, $target_encoding, $encoding);
// remove any question marks that have been introduced because of illegal characters
$target = str_replace( "?", "", $target );
// replace the token string "[question_mark]" with the symbol "?"
$target = str_replace( "[question_mark]", "?", $target );
return $target;
}
?>
Hope this helps someone! (Admins should feel free to delete my previous, incorrect, post for clarity)
-AIf you are trying to generate a CSV (with extended chars) to be opened at Exel for Mac, the only that worked for me was:
<?php mb_convert_encoding( $CSV, 'Windows-1252', 'UTF-8'); ?>
I also tried this:
<?php
//Separado OK, chars MAL
iconv('MACINTOSH', 'UTF8', $CSV);
//Separado MAL, chars OK
chr(255).chr(254).mb_convert_encoding( $CSV, 'UCS-2LE', 'UTF-8');
?>
But the first one didn't show extended chars correctly, and the second one, did't separe fields correctlyTo add to the Flash conversion comment below, here's how I convert back from what I've stored in a database after converting from Flash HTML text field output, in order to load it back into a Flash HTML text field:
function htmltoflash($htmlstr)
{
return str_replace("<br />","\n",
str_replace("<","<",
str_replace(">",">",
mb_convert_encoding(html_entity_decode($htmlstr),
"UTF-8","ISO-8859-1"))));
}Note that `mb_convert_encoding($val, 'HTML-ENTITIES')` does not escape '\'', '"', '<', '>', or '&'.It appears that when dealing with an unknown "from encoding" the function will both throw an E_WARNING and proceed to convert the string from ISO-8859-1 to the "to encoding".When you need to convert from HTML-ENTITIES, but your UTF-8 string is partially broken (not all chars in UTF-8) - in this case passing string to mb_convert_encoding($string, 'UTF-8', 'HTML-ENTITIES'); - corrupts chars in string even more. In this case you need to replace html entities gradually to preserve character good encoding. I wrote such closure for this job :
<?php
$decode_entities = function($string) {
preg_match_all("/&#?\w+;/", $string, $entities, PREG_SET_ORDER);
$entities = array_unique(array_column($entities, 0));
foreach ($entities as $entity) {
$decoded = mb_convert_encoding($entity, 'UTF-8', 'HTML-ENTITIES');
$string = str_replace($entity, $decoded, $string);
}
return $string;
};
?>Clean a string for use as filename by simply replacing all unwanted characters with underscore (ASCII converts to 7bit). It removes slightly more chars than necessary. Hope its useful.
$fileName = 'Test:!"$%&/()=ÖÄÜöäü<<';
echo strtr(mb_convert_encoding($fileName,'ASCII'),
' ,;:?*#!§$%&/(){}<>=`´|\\\'"',
'____________________________');/**
* Convert Windows-1250 to UTF-8
* Based on https://www.php.net/manual/en/function.mb-convert-encoding.php#112547
*/
class TextConverter
{
private const ENCODING_TO = 'UTF-8';
private const ENCODING_FROM = 'ISO-8859-2';
private array $mapChrChr = [
0x8A => 0xA9,
0x8C => 0xA6,
0x8D => 0xAB,
0x8E => 0xAE,
0x8F => 0xAC,
0x9C => 0xB6,
0x9D => 0xBB,
0xA1 => 0xB7,
0xA5 => 0xA1,
0xBC => 0xA5,
0x9F => 0xBC,
0xB9 => 0xB1,
0x9A => 0xB9,
0xBE => 0xB5,
0x9E => 0xBE
];
private array $mapChrString = [
0x80 => '€',
0x82 => '‚',
0x84 => '„',
0x85 => '…',
0x86 => '†',
0x87 => '‡',
0x89 => '‰',
0x8B => '‹',
0x91 => '‘',
0x92 => '’',
0x93 => '“',
0x94 => '”',
0x95 => '•',
0x96 => '–',
0x97 => '—',
0x99 => '™',
0x9B => '’',
0xA6 => '¦',
0xA9 => '©',
0xAB => '«',
0xAE => '®',
0xB1 => '±',
0xB5 => 'µ',
0xB6 => '¶',
0xB7 => '·',
0xBB => '»'
];
/**
* @param $text
* @return string
*/
public function execute($text): string
{
$map = $this->prepareMap();
return html_entity_decode(
mb_convert_encoding(strtr($text, $map), self::ENCODING_TO, self::ENCODING_FROM),
ENT_QUOTES,
self::ENCODING_TO
);
}
/**
* @return array
*/
private function prepareMap(): array
{
$maps[] = $this->arrayMapAssoc(function ($k, $v) {
return [chr($k), chr($v)];
}, $this->mapChrChr);
$maps[] = $this->arrayMapAssoc(function ($k, $v) {
return [chr($k), $v];
}, $this->mapChrString);
return array_merge([], ...$maps);
}
/**
* @param callable $function
* @param array $array
* @return array
*/
private function arrayMapAssoc(callable $function, array $array): array
{
return array_column(
array_map(
$function,
array_keys($array),
$array
),
1,
0
);
}
}instead of ini_set(), you can try this
mb_substitute_character("none");Another sample of recoding without MultiByte enabling.
(Russian koi->win, if input in win-encoding already, function recode() returns unchanged string)
<?php
// 0 - win
// 1 - koi
function detect_encoding($str) {
$win = 0;
$koi = 0;
for($i=0; $i<strlen($str); $i++) {
if( ord($str[$i]) >224 && ord($str[$i]) < 255) $win++;
if( ord($str[$i]) >192 && ord($str[$i]) < 223) $koi++;
}
if( $win < $koi ) {
return 1;
} else return 0;
}
// recodes koi to win
function koi_to_win($string) {
$kw = array(128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 254, 224, 225, 246, 228, 229, 244, 227, 245, 232, 233, 234, 235, 236, 237, 238, 239, 255, 240, 241, 242, 243, 230, 226, 252, 251, 231, 248, 253, 249, 247, 250, 222, 192, 193, 214, 196, 197, 212, 195, 213, 200, 201, 202, 203, 204, 205, 206, 207, 223, 208, 209, 210, 211, 198, 194, 220, 219, 199, 216, 221, 217, 215, 218);
$wk = array(128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 225, 226, 247, 231, 228, 229, 246, 250, 233, 234, 235, 236, 237, 238, 239, 240, 242, 243, 244, 245, 230, 232, 227, 254, 251, 253, 255, 249, 248, 252, 224, 241, 193, 194, 215, 199, 196, 197, 214, 218, 201, 202, 203, 204, 205, 206, 207, 208, 210, 211, 212, 213, 198, 200, 195, 222, 219, 221, 223, 217, 216, 220, 192, 209);
$end = strlen($string);
$pos = 0;
do {
$c = ord($string[$pos]);
if ($c>128) {
$string[$pos] = chr($kw[$c-128]);
}
} while (++$pos < $end);
return $string;
}
function recode($str) {
$enc = detect_encoding($str);
if ($enc==1) {
$str = koi_to_win($str);
}
return $str;
}
?>// convert UTF8 to DOS = CP850
//
// $utf8_text=UTF8-Formatted text;
// $dos=CP850-Formatted text;
// have fun
$dos = mb_convert_encoding($utf8_text, "CP850", mb_detect_encoding($utf8_text, "UTF-8, CP850, ISO-8859-15", true));Why did you use the php html encode functions? mbstring has it's own Encoding which is (as far as I tested it) much more usefull:
HTML-ENTITIES
Example:
$text = mb_convert_encoding($text, 'HTML-ENTITIES', "UTF-8");If you are attempting to convert "UTF-8" text to "ISO-8859-1" and the result is always returning in "ASCII", place the following line of code before the mb_convert_encoding:
mb_detect_order(array('UTF-8', 'ISO-8859-1'));
It is necessary to force a specific search order for the conversion to workFor those wanting to convert from $set to MacRoman, use iconv():
<?php
$string = iconv('UTF-8', 'macintosh', $string);
?>
('macintosh' is the IANA name for the MacRoman character set.)As an alternative to Johannes's suggestion for converting strings from other character sets to a 7bit representation while not just deleting latin diacritics, you might try this:
<?php
$text = iconv($from_enc, 'US-ASCII//TRANSLIT', $text);
?>
The only disadvantage is that it does not convert "ä" to "ae", but it handles punctuation and other special characters better.
--
DavidWhen converting Japanese strings to ISO-2022-JP or JIS on PHP >= 5.2.1, you can use "ISO-2022-JP-MS" instead of them.
Kishu-Izon (platform dependent) characters are converted correctly with the encoding, as same as with eucJP-win or with SJIS-win.be careful when converting from iso-8859-1 to utf-8.
even if you explicitly specify the character encoding of a page as iso-8859-1(via headers and strict xml defs), windows 2000 will ignore that and interpret it as whatever character set it has natively installed.
for example, i wrote char #128 into a page, with char encoding iso-8859-1, and it displayed in internet explorer (& mozilla) as a euro symbol.
it should have displayed a box, denoting that char #128 is undefined in iso-8859-1. The problem was it was displaying in "Windows: western europe" (my native character set).
this led to confusion when i tried to convert this euro to UTF-8 via mb_convert_encoding()
IE displays UTF-8 correctly- and because PHP correctly converted #128 into a box in UTF-8, IE would show a box.
so all i saw was mb_convert_encoding() converting a euro symbol into a box. It took me a long time to figure out what was going on.rodrigo at bb2 dot co dot jp wrote that inconv works better than mb_convert_encoding, I find that when converting from uft8 to shift_jis
$conv_str = mb_convert_encoding($str,$toCS,$fromCS);
works while
$conv_str = iconv($fromCS,$toCS.'//IGNORE',$str);
removes tildes from $str.For those who can´t use mb_convert_encoding() to convert from one charset to another as a metter of lower version of php, try iconv().
I had this problem converting to japanese charset:
$txt=mb_convert_encoding($txt,'SJIS',$this->encode);
And I could fix it by using this:
$txt = iconv('UTF-8', 'SJIS', $txt);
Maybe it´s helpfull for someone else! ;)If you want to convert japanese to ISO-2022-JP it is highly recommended to use ISO-2022-JP-MS as the target encoding instead. This includes the extended character set and avoids ? in the text. For example the often used "1 in a circle" ① will be correctly converted then.Here's a tip for anyone using Flash and PHP for storing HTML output submitted from a Flash text field in a database or whatever.
Flash submits its HTML special characters in UTF-8, so you can use the following function to convert those into HTML entity characters:
function utf8html($utf8str)
{
return htmlentities(mb_convert_encoding($utf8str,"ISO-8859-1","UTF-8"));
}I\'d like to share some code to convert latin diacritics to their
traditional 7bit representation, like, for example,
- à,ç,é,î,... to a,c,e,i,...
- ß to ss
- ä,Ä,... to ae,Ae,...
- ë,... to e,...
(mb_convert \"7bit\" would simply delete any offending characters).
I might have missed on your country\'s typographic
conventions--correct me then.
<?php
/**
* @args string $text line of encoded text
* string $from_enc (encoding type of $text, e.g. UTF-8, ISO-8859-1)
*
* @returns 7bit representation
*/
function to7bit($text,$from_enc) {
$text = mb_convert_encoding($text,\'HTML-ENTITIES\',$from_enc);
$text = preg_replace(
array(\'/ß/\',\'/&(..)lig;/\',
\'/&([aouAOU])uml;/\',\'/&(.)[^;]*;/\'),
array(\'ss\',\"$1\",\"$1\".\'e\',\"$1\"),
$text);
return $text;
}
?>
Enjoy :-)
Johannes
==
[EDIT BY danbrown AT php DOT net: Author provided the following update on 27-FEB-2012.]
==
An addendum to my "to7bit" function referenced below in the notes.
The function is supposed to solve the problem that some languages require a different 7bit rendering of special (umlauted) characters for sorting or other applications. For example, the German ß ligature is usually written "ss" in 7bit context. Dutch ÿ is typically rendered "ij" (not "y").
The original function works well with word (alphabet) character entities and I've seen it used in many places. But non-word entities cause funny results:
E.g., "©" is rendered as "c", "­" as "s" and "&rquo;" as "r".
The following version fixes this by converting non-alphanumeric characters (also chains thereof) to '_'.
<?php
/**
* @args string $text line of encoded text
* string $from_enc (encoding type of $text, e.g. UTF-8, ISO-8859-1)
*
* @returns 7bit representation
*/
function to7bit($text,$from_enc) {
$text = preg_replace(/W+/,'_',$text);
$text = mb_convert_encoding($text,'HTML-ENTITIES',$from_enc);
$text = preg_replace(
array('/ß/','/&(..)lig;/',
'/&([aouAOU])uml;/','/ÿ/','/&(.)[^;]*;/'),
array('ss',"$1","$1".'e','ij',"$1"),
$text);
return $text;
}
?>
Enjoy again,
Johannes