Palantir and AOL workaround

mepbmlist
#1

As a lot of people are discovering, AOL mangles the XML files so that
Palantir cannot import them.

Thanks to some volunteers from my team, I've worked out what exactly
is happening to the files, and I have a workaround which will allow
people to edit their XML files so that Palantir can import them. I've
only worked out exactly how to do this on Windows, so maybe some Mac
techie can work out how to do it there. Basically, we want to insert
the bytes FEFF at the very beginning of the XML file.

Unfortunately, this edit prevents Automagic from importing the XML
files, so make sure you either import your XML into Automagic before
doing this, or make a backup copy for use with Automagic first!

To make XML files downloaded through AOL usable by Palantir:
1) Open the XML file with Wordpad. Do NOT use Notepad.
    You should see a bunch of undisplayable characters before each letter.
2) At the very beginning of the file, before the first undisplayable
character, use the numeric keypad to enter two special characters.
    To do this, hold down alt, then type in on the numeric keypad
"0254". Then let up alt. You should see a thorn character (looking
rather like a P with the loop halfway up the shaft rather than at the
top of the shaft) appear. After the thorn, hold down alt again and
type "0255". This time you should get a lower case y with two dots
over it.
3) Save the file. At this point, Palantir should be able to read it.

Don't do this for files that didn't come through AOL, or for files
that don't show the boxes indicating undisplayable characters before
each letter.

You may not be able to load the file in Wordpad anymore - when I do
this it allows Wordpad to recognize the encoding as UTF-16, which it
knows it can't handle. So it's hard to go back.

For those interested in the technical details - what's happening is
that for some reason AOL is changing the encoding of the XML document
from UTF-8 to UTF-16. However, they're omitting the byte order marker
which is supposed to begin any UTF-16 XML document. The process above
adds that byte order marker, which is all Palantir needs to identify
the document as UTF-16 rather than UTF-8. Unfortunately, Automagic
gets thrown off by the byte order marker, and can't import the XML
anymore.

Hope this proves helpful to those stuck with AOL. If anyone is
interested I can also describe how to convert AOL files using Emacs,
my editor of choice.

-Peter

#2

Peter, any idea if you can forward the AOL email to a
web-based account to get around this?

Yahoo also mangles xml files, so I now forward my turn
results to a dummy Hotmail account and download them
from there.

Ok, I'm catching up and see Palantir is now available
to all ME gamers, I've got the current version, and it
works fine to skip a few turns.

Cheers,

Dan

···

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

#3

Peter, any idea if you can forward the AOL email to a
web-based account to get around this?

As I understand AOL's process, no - the problem is that they zip
together the incoming .pdf and .xml files before they reach the user,
and something about their .zip process changes the encoding to UTF-16.
Once the email has hit AOL, it's no longer usable without
modification. But I don't have an AOL account, so I can't test this
myself.

Yahoo also mangles xml files, so I now forward my turn
results to a dummy Hotmail account and download them
from there.

Does Yahoo actually alter the XML, or does it just render it as part
of the web page? My old web mail provider insisted on displaying the
XML as part of the message, which was annoying but could be dealt with
- you just change your browser's encoding setting to UTF-8, copy out
the XML, paste it into a text document, and delete the extra return
characters. Not painless but not quite as obscure as the AOL encoding
change.

-Peter

···

--- In mepbmlist@yahoogroups.com, D N <nanooknw@y...> wrote:

#4

Does Yahoo actually alter the XML, or does it just

render it as part
of the web page? My old web mail provider insisted on
displaying the
XML as part of the message, which was annoying but
could be dealt with
- you just change your browser's encoding setting to
UTF-8, copy out
the XML, paste it into a text document, and delete the
extra return
characters. Not painless but not quite as obscure as
the AOL encoding
change.

This is how I've been dealing with it until recently,
but I found the latest version of Automagic to be more
finicky about what it would accept. Pasting directly
from Yahoo changes some characters with unusual accent
marks. It just became easier to forward the turn and
get the xml directly, so that's all I've done for the
last couple months.

As I understand AOL's process, no - the problem is

that they zip
together the incoming .pdf and .xml files before they
reach the user,
and something about their .zip process changes the
encoding to UTF-16.
Once the email has hit AOL, it's no longer usable
without
modification.

That's a shame. Perhaps some AOL users would like to
setup a web-based account (at Hotmail or anyplace that
doesn't alter the xml) to receive turns.

Dan

···

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

#5

That's caused by encoding problems - you should be able to fix the
accent marks by changing your browser to UTF-8 before copying the XML.

On IE, use the Encoding option under the View menu, and select Unicode
(UTF-8). The accented characters should then render correctly in your
browser window, and copy and paste correctly too.

-Peter

···

--- In mepbmlist@yahoogroups.com, D N <nanooknw@y...> wrote:

This is how I've been dealing with it until recently,
but I found the latest version of Automagic to be more
finicky about what it would accept. Pasting directly
from Yahoo changes some characters with unusual accent
marks. It just became easier to forward the turn and
get the xml directly, so that's all I've done for the
last couple months.

#6

Thanks, Peter. It does work, with forcing Notepad to
save the file as xml.

Dan

Pasting directly

from Yahoo changes some characters with unusual

accent

marks.

That's caused by encoding problems - you should be

able to fix the
accent marks by changing your browser to UTF-8 before
copying the XML.
On IE, use the Encoding option under the View menu,
and select Unicode
(UTF-8). The accented characters should then render
correctly in your
browser window, and copy and paste correctly too.

···

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

#7

Does Yahoo actually alter the XML, or does it just render it as part
of the web page? My old web mail provider insisted on displaying the
XML as part of the message, which was annoying but could be dealt with
- you just change your browser's encoding setting to UTF-8, copy out
the XML, paste it into a text document, and delete the extra return
characters. Not painless but not quite as obscure as the AOL encoding
change.

How would you do that? Can you put this in the XML informing the program that reads it to display it in a specific format. BTW thanks very much for all your assistance here - excellent advice.

Clint

···

-Peter

Middle Earth PBM - hit reply to send to everyone
To Unsubscribe: http://www.yahoogroups.com
Website: http://www.MiddleEarthGames.com

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

****************************************************************
                    Middle Earth Games
Mailto: me@middleearthgames.com
Website: www.middleearthgames.com

UK: 340 North Road, Cardiff CF14 3BP UK
US: EpicMail, 12330 Perry Highway, 2nd Floor, Wexford, PA15090

Phone Times: 10am-6.30pm UK Time (BST);5am-1.30 (EST)
UK: 029 2091 3359 (029 2062 5665 can be used if main is engaged)
(Dial 011 44 2920 913359 if in the US)
UK Fax: 029 2062 5532 24 hours
US Fax: 1-503-296-2325 (preferred)
US Phone: 724 933 3132 EST Weekdays
US alternate Fax: 801 650 8073 Fax 24hrs
****************************************************************

#8

>Does Yahoo actually alter the XML, or does it just render it as part
>of the web page? My old web mail provider insisted on displaying the
>XML as part of the message, which was annoying but could be dealt with
>- you just change your browser's encoding setting to UTF-8, copy out
>the XML, paste it into a text document, and delete the extra return
>characters. Not painless but not quite as obscure as the AOL encoding
>change.

How would you do that?

Each browser has its own mechanism for specifying the encoding for the
current page. In Internet Explorer, there's an Encoding option under
the View dropdown menu, which should have Unicode (UTF-8) as one of
its options. People may need to expand the More option to see it,
though. In Mozilla, the View dropdown has a Character Coding option,
which should also offer Unicode (UTF-8), possibly under More. I'm not
sure about Netscape 6.0, but it should work more or less like Mozilla.

This next bit is moderately technical, and of interest to ME Games and
maybe Automagic developers only.

Can you put this in the XML informing the program
that reads it to display it in a specific format.

Yes, but I'm not sure it'll help - the problem here is that the
programs that are mangling the XML (AOL's mail software and the web
mail clients) aren't actually reading the XML as XML. Only Automagic
and Palantir are doing that.

After a little more investigation, it looks to me like the best way to
try to resolve this will require some information on your email
software - currently, it looks like your mail software is sending the
.xml files as MIME type text/xml. This is basically correct, but I
think you might have more luck with the MIME type set to
application/xml. Web mail clients probably treat application/xml the
same way they treat application/pdf - offering the user a method to
download it directly rather than presenting it as text. AOL's
behavior is harder to predict, but I'd hope they don't change anything
about application/* attachments.

Also, you're supposed to be able to specify a "charset" attribute in
your MIME header for either type text/xml or application/xml. You
might also have better luck with AOL and possibly with the web mail
clients if you can specify that that's UTF-8. I suspect that just
changing to application/xml would be sufficient, though.

I have no idea how to actually make any of this happen, though - it's
entirely dependent on your email software. It could be easy, or so
hard as to be practically impossible.

If you do end up explicitly setting the coding in the XML document,
the mechanism for doing so is to add an XML declaration to the very
beginning of the file:
<?xml version="1.0" encoding="utf-8"?>

But I know Automagic isn't expecting that to be there, and would need
to be updated to allow it.

Another option might be to generate the XML in the iso-8859-1
encoding. I don't know if this is a good idea at all, and I
personally wouldn't even try unless I knew I had no way to control the
MIME type or that had failed to solve the problem, but it shouldn't be
too hard to make it happen - I don't know how your software is
creating the XML, but it's trivial in XSL, for instance. Doing so
would require an encoding declaration at the beginning of the file, so
again Automagic at least would require an update. It might also
require an update to recognize the iso-8859-1 encoding too. I'm
relatively confident that Palantir would not - it looks like Palantir
is using a full XML parser, which should recognize iso-8859-1 just
fine as long as it's told to look for it.

The declaration there would look like:
<?xml version="1.0" encoding="iso-8859-1"?>

It's likely but not certain that doing that would prevent AOL from
mangling the XML file. It should definitely help people using web
mail accounts who have to copy and paste the XML.

-Peter

···

--- In mepbmlist@yahoogroups.com, Middle Earth PBM Games <me@M...> wrote:

#9

Peter, thanks for all the info -- helpful even to read
as a player.

Another thought about AOL (and Yahoo): in a previous
post, you suggested a workaround from the player's
end, but noted it was cumbersome -- many steps to
correct the text changes.

Would it be possible for someone to automate that?
Create a small program to fix the corrupt information?
Scott Moyes and I used several, routine steps to fix
the altered text from Yahoo, and I wonder if that
could be done by a mini program.

Your latest email suggested a method MEGames could try
to prevent AOL from mangling the files, and if that
works, great. But if not, perhaps it's possible to fix
at the user's end.

Dan

···

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

#10

As soon as clints decided what he wants to do with regard zipping the xml,
ill try a few things to make it work with aol et al (forcing the charset
etc)

···

----- Original Message -----
From: "D N" <nanooknw@yahoo.com>
To: <mepbmlist@yahoogroups.com>
Sent: Tuesday, September 30, 2003 7:55 PM
Subject: Re: [mepbmlist] Re: Palantir and AOL workaround

Peter, thanks for all the info -- helpful even to read
as a player.

Another thought about AOL (and Yahoo): in a previous
post, you suggested a workaround from the player's
end, but noted it was cumbersome -- many steps to
correct the text changes.

Would it be possible for someone to automate that?
Create a small program to fix the corrupt information?
Scott Moyes and I used several, routine steps to fix
the altered text from Yahoo, and I wonder if that
could be done by a mini program.

Your latest email suggested a method MEGames could try
to prevent AOL from mangling the files, and if that
works, great. But if not, perhaps it's possible to fix
at the user's end.

Dan

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

Middle Earth PBM - hit reply to send to everyone
To Unsubscribe: http://www.yahoogroups.com
Website: http://www.MiddleEarthGames.com

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

#11

Peter, thanks for all the info -- helpful even to read
as a player.

Another thought about AOL (and Yahoo): in a previous
post, you suggested a workaround from the player's
end, but noted it was cumbersome -- many steps to
correct the text changes.

Once you've done it once or twice it shouldn't be too bad. At least
for the AOL problem. Open it, type in the eight digits, close it again.

Would it be possible for someone to automate that?
Create a small program to fix the corrupt information?
Scott Moyes and I used several, routine steps to fix
the altered text from Yahoo, and I wonder if that
could be done by a mini program.

Absolutely. All you have to do to make the AOL-mangled XML readable
by Palantir is add a specific two byte sequence to the very beginning
of the file. Any programming language should be able to do that with
no trouble; I could do it as a Perl one-liner. The problem is getting
it into a cross-platform tool that doesn't make any unsafe assumptions
about the user's environment or require installation of significant
new tools. That's not all that much of a problem, but it's more work
than I'm really up for right now.

If anyone wants to volunteer to try to develop something along these
lines, I'll certainly explain what I found during my investigations
and what I believe needs to be done.

-Peter

···

--- In mepbmlist@yahoogroups.com, D N <nanooknw@y...> wrote:

#12

Peter, I've just tried the suggested workaround for an
AOL corrupted XML file (pasted below), and it's now
mostly readable with IE, but near the end I get

<CommanderTThe XML page cannot be displayed

which, I assume, means there is another place in the
file that needs editing.

When I open it in Wordpad, I see boxes for unreadable
characters and then two long blank sections.

Is there a way I can complete the corrections to make
the file usable by Palantir?

Thanks,

Dan

1) Open the XML file with Wordpad. Do NOT use Notepad.
You should see a bunch of undisplayable characters
before each letter.
2) At the very beginning of the file, before the first
undisplayable
character, use the numeric keypad to enter two special
characters.
To do this, hold down alt, then type in on the numeric
keypad
"0254". Then let up alt. You should see a thorn
character (looking
rather like a P with the loop halfway up the shaft
rather than at the
top of the shaft) appear. After the thorn, hold down
alt again and
type "0255". This time you should get a lower case y
with two dots
over it.
3) Save the file. At this point, Palantir should be
able to read it.

···

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

#13

Peter, I've just tried the suggested workaround for an
AOL corrupted XML file (pasted below), and it's now
mostly readable with IE, but near the end I get

<CommanderTThe XML page cannot be displayed

Hmm. Does the file correctly import into Palantir? I don't know if
IE is as good at detecting the UTF-16 encoding that AOL uses as
Palantir. It _should_ be but you never know.

How does it compare in size to the pre-edit copy? Are the blank
sections you mentioned present in the pre-edit copy?

Is there a way I can complete the corrections to make
the file usable by Palantir?

I don't know of any other corrections which need to be made; the
changes I described worked for my test case XML. If you want you can
forward the .xml file to my email (pdeglopper(at)geekmail.cc) and I'll
take a look. I'm currently playing in games 223 and 136 (War of the
Ring and 1650 Gunboat) so if it's one a turn for of those games that's
not really an option, unfortunately.

-Peter

···

--- In mepbmlist@yahoogroups.com, D N <nanooknw@y...> wrote:

#14

Hmm. Does the file correctly import into Palantir?

No.

I don't know if

IE is as good at detecting the UTF-16 encoding that
AOL uses as
Palantir. It _should_ be but you never know.

After inserting the new characters with Wordpad, I can
at least view the start of the file with IE. Before
editing it, all I got was an error message.

How does it compare in size to the pre-edit copy?

Pre-edit, 61kb. Edited, 63.

Are the blank

sections you mentioned present in the pre-edit copy?

I believe so, but it's hard to say. The pre-edit copy
shows, in Wordpad, a series of squares mixed in with
letters and other characters, but also what looks liek
two blank sections. The edited copy is all squares,
with those blank sections. They display differently,
so it's tough to compare.

I'll forward it. Thanks for the look!

Dan

···

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

#15

I found the problem; I left a step out of the conversion I wrote up on
Friday.

So this is even more tedious than before - with any luck this will be
a very temporary solution! But it does seem to work. Apologies for
the complicated nature of the workaround, I haven't been able to find
anything simpler that uses commonly available tools.

The remaining problem, besides the need to insert the two characters
at the beginning of the file as I described Friday, is that the return
characters in the file are not encoded correctly either. So AOL has
seriously ruined the file - I have no idea what their software is
trying to do.

In order to make it work, you need to take out all the line feeds in
the AOL generated file. To do this, during the same Wordpad session
in which you add the special characters:

Move the curser so it's to the left of the box shown all the way at
the left of the word Economy - this should look something like this,
if you imagine the box characters instead of periods:

.<.E.c.o.n.o.m.y.>

It should stand out, because there'll be a blank line with just a box
immediately above it.

Once you're at the beginning of that line, hit backspace exactly four
times.

It should merge the economy line onto the characters line. At this
point, if you move the curser you should see:
.<./.C.h.a.r.a.c.t.e.r.s.>.<.E.c.o.n.o.m.y.>
with exactly one box between the > and < characters.

Then do the same for the Pop Centers line, merging it on to the end of
the Economy line.

Then move all the way to the end of the file - hitting page down
should do it. Hit backspace four times again, so that the file ends
with a > character.

Do this before or after adding the two characters at the beginning of
the file.

Once you do this, IE should be able to read the file and Palantir
should be able to import it.

I'd appreciate it if anyone who actually goes to the trouble of doing
this (and I can understand why you wouldn't want to) would let me know
whether it worked for them or not.

-Peter

···

--- In mepbmlist@yahoogroups.com, D N <nanooknw@y...> wrote:

>Hmm. Does the file correctly import into Palantir?

No.

#16

Removing all the 'line feeds' (or 'paragraph marks') may fix things just
fine for Palantir, but Automagic will start to have fits with this. The
character info should probably import fine, but Automagic looks for
those 'line feeds' to determine the different sections of the XML file
in order to sort it into manageable chunks. The 1st section is
nation/character info, the 2nd is economy, etc.

Mike Mulka

In order to make it work, you need to take out all the line feeds in
the AOL generated file. To do this, during the same Wordpad session
in which you add the special characters:

Move the curser so it's to the left of the box shown all the way at
the left of the word Economy - this should look something like this,
if you imagine the box characters instead of periods:

.<.E.c.o.n.o.m.y.>

It should stand out, because there'll be a blank line with just a box
immediately above it.

Once you're at the beginning of that line, hit backspace exactly four
times.

It should merge the economy line onto the characters line. At this
point, if you move the curser you should see:
.<./.C.h.a.r.a.c.t.e.r.s.>.<.E.c.o.n.o.m.y.>
with exactly one box between the > and < characters.

[Non-text portions of this message have been removed]

#17

Oh, none of these edits to the files to make Palantir able to import
the files work at all with Automagic. Just adding the special
characters (the initial attempt at a fix I described Friday) makes the
file unreadable by Automagic. Fortunately, the raw AOL files seem to
be importable by Automagic, so as long as you keep a pre-edit copy or
do the Automagic import prior to making the edits it should be ok.

-Peter

···

--- In mepbmlist@yahoogroups.com, "Urzahil" <urzahil@d...> wrote:

Removing all the 'line feeds' (or 'paragraph marks') may fix things just
fine for Palantir, but Automagic will start to have fits with this. The
character info should probably import fine, but Automagic looks for
those 'line feeds' to determine the different sections of the XML file
in order to sort it into manageable chunks. The 1st section is
nation/character info, the 2nd is economy, etc.

Mike Mulka

#18

Move the curser so it's to the left of the box... let

me know
whether it worked for them or not.

This does work, even with the feisty file I described
earlier. Thanks, Peter. It makes the file display
normally as xml in IE, and Palantir can read it just
fine.

It's also not that complicated once you make sense of
the strange display in Wordpad, but it is annoying to
have to do it. If a file ended up needing many more
such corrections, it'd be cumbersome.

Mark, is it possible (and reasonable) to consider
automating this with Palantir?

Thanks,

Dan

···

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

#19

Mark, is it possible (and reasonable) to consider
automating this with Palantir?

We're working on it. AOL pain in the....

Clint

#20

My Notepad method of fixing AOL-corrupted XML files, posted on the
forum, does allow both Automagic and Palantir to import the XML.
The only problem I foresee is since my fix requires the replacement
of special letter characters with their normal equivalent, Automagic
imports character names exactly as you make them in the XML. I
don't know if orders sent in with Automagic have to have the special
letters in the character name. Clint? This could be a problem for
characters like Murazor, where a special letter should appear in the
first five letters. Other than this potential problem, the fix
works great (confirmed on WinXP and 2000 systems at any rate).

Dustin

--- In mepbmlist@yahoogroups.com, "pdeglopper2000" <pdeglopper@g...>
wrote:

> Removing all the 'line feeds' (or 'paragraph marks') may fix

things just

> fine for Palantir, but Automagic will start to have fits with

this. The

> character info should probably import fine, but Automagic looks

for

> those 'line feeds' to determine the different sections of the

XML file

> in order to sort it into manageable chunks. The 1st section is
> nation/character info, the 2nd is economy, etc.
>
> Mike Mulka

Oh, none of these edits to the files to make Palantir able to

import

the files work at all with Automagic. Just adding the special
characters (the initial attempt at a fix I described Friday) makes

the

file unreadable by Automagic. Fortunately, the raw AOL files seem

to

be importable by Automagic, so as long as you keep a pre-edit copy

or

···

--- In mepbmlist@yahoogroups.com, "Urzahil" <urzahil@d...> wrote:
do the Automagic import prior to making the edits it should be ok.

-Peter