moderated, Malformed utf-8 not caught by Encode

1. Reading Text File Encoding and converting to Perls internal UTF-8 encoding

Need help from Unicode guru's or anybody with some knowledge on the subject.

I maybe have a text (character) file I just open. But I don't know the encoding and I
can't open it with any encoding attribute.

It would appear to me that at the start of the file, there is an encoding mark (or none),
assuming a text file, a sort of BOM sequence of octets that mark what its encoding is.

Given that I might be passed a file descriptor only, I am module, and I rewind the position
to the start of the file, is there any  way I can tell the encoding. If I could, and
its not utf8, I could decode() the rest of the file as octets, ie: in-place memeory decode,
create a temp file decoded, or possibly re-open it with the proper encoding.

I think that encoding is the usual 8/16/32 bit utf but with many locales (chars).

I am still sketchy where to find a list of encoding markers to be able to find out
this information. And still sketchy on the methods available for analysis and transformation.

I know Perl has a massive 'use Encode' lib, nevertheless, this is what I need to do to finalize
a module I'm working on.

Thanks for the help.
-sln

2. UTF - SEEK_SET workaround for BOM encoding(utf-16/32) layer Bug - Perl

3. Malformed UTF-8 character

I get a nasty error...
Malformed UTF-8 character (unexpected non-continuation byte 0x02,
immediately af
ter start byte 0xe7) in null operation at C:/Perl/site/lib/Tk.pm line
406.
This is my code:
my @dbnames;
my $dbhost  = "localhost";
my $dbuser  = "testuser";
my $dbpass  = "logpass";
my($dbh,$sth,$query);
my $dsn = "DBI:mysql:host=$dbhost";
$dbh    = DBI->connect($dsn,$dbuser,$dbpass,{PrintError => 0, RaiseError
=> 1});
$query  = qq^SHOW DATABASES^;
$sth    = $dbh->prepare($query);
$sth->execute();
while(my $ref_db = $sth->fetchrow_array()) {
    print "Found database: $ref_db\n";
    push(@dbnames, $ref_db);
}
$sth->finish();
$dbh->disconnect();
$i=0;
foreach(@dbnames){
  $w=$pane ->Label(-text=>"$_", -relief=>'groove', -width=>30);
  $pane->windowCreate('end', -window=>$w);
  $ch_t = $w->cget(-text);
  print "$ch_t\n";
  $w2 = $pane->Checkbutton(-bg=>'white',
-text=>\$dbnames[$i],-command=>sub{print "Check
$dbnames[$i]\n"})->pack(-anchor=>'nw');
  $pane->windowCreate('end', -window=> $w2);
  $pane->insert('end',"\n");
  $i++;
}
$pane->configure(-state=>'disabled');
-----------------------------------------------------------
I get the error because of this: -text=>\$dbnames[$i]
I've looked at utf8 and use Encode... but didn't solved this error.

4. Meaning of "Malformed UTF-8 character"? - Perl

5. encoding - utf-8

Hello,

I have a problem, apparently on an encoding issue, but can't figure out 
where it comes from. Could someone please help?

I'm reading from an XML file that contains the line

[1]	...Bergson referred as "dur"; the way...

Then I parse the file with XML::DOM::Parser and print it out again.
The line now becomes:

[2]	...Bergson referred as "dur㩥; the way...


Where can this possibly come from? Does "standard" reading and printing 
not produce UTF-8? And does XML::DOM::Parser not read input as UTF-8? 
So, when I print it out, should it not be UTF-8 again?

The file containing the first line was written like this:

	#!/usr/bin/perl
	use strict;
	use warnings;
	use encoding 'utf-8';

	my $infile = "file1.xml";
	open IN, "$infile" or die "\ncannot read specified infile\n";
	my $text = join "", <IN>;
	close IN;

	# some processing...

	my $outfile = "file2.xml";
	open OUT, ">:encoding(utf-8)", $outfile or die "cannot create out file";
	print OUT $text;
	close OUT;

	# alternatively I tried:
	# open IN, "<:encoding(utf-8)", "$infile"; # and
	# open OUT, ">$outfile" or die "cannot create out file";
	# respectively. It makes no difference.


The second script reads/writes like this:

	#!/usr/bin/perl
	use strict;
	use XML::DOM;
	use warnings;

	my $infile = "file2.xml";
	my $dom_parser = new XML::DOM::Parser();
	my $TREE = $dom_parser->parsefile($infile);

	open OUT, ">file3.xml" or die "could not open log file";
	print OUT $TREE->toString();
	close OUT;


Thanks for any comments!

Alois Heuboeck

6. XML::Twig produces double encoded UTF-8

7. LWP and UTF-8 encoding

I have a short perl script that among other things tries to open web
pages using LWP::UserAgent.

here is an excerpt from the code:

my $ua = LWP::UserAgent->new;

$ua->default_headers->push_header('Accept-Charset' =>
"iso-8859-1,iso-8859-2,utf-8");
$ua->default_headers->push_header('Accept' => "text/html, text/plain,
image/*");

$req = new HTTP::Request('GET', $url);
$res = $ua->request($req);

I'm using $res->content to get the content of the retrieved page.
Sometimes the script produces the following warnings with some Web
sites. I don't know why i'm getting this kind of messages.

Parsing of undecoded UTF-8 will give garbage when decoding entities at
/usr/lib/perl5/vendor_perl/5.8.6/LWP/Protocol.pm line 114.
Parsing of undecoded UTF-8 will give garbage when decoding entities at
/usr/lib/perl5/site_perl/5.8.6/i386-linux-thread-multi/HTML/PullParser.pm
line 83.

8. Question about Encode (Windows-1252 to utf-8) - Perl