There is an error in XML Document (1, nnnnnn)

nickmoyle · Post by **nickmoyle** » Thu Aug 30, 2007 7:04 pm

Hi, myself and a colleague are having issues with Vault 4 running on Vista. We have several other colleagues on a different site also running Vault on Vista, and it is working fine for them. The people Vault is operating fine for are those working at the same site as the Vault server, so they have local network access. We are working remotely.

The above error occurs after log in, and once a repository is selected. We have about 8 repositories. 1 of them we can get to, the others produce the above error (title). The 1 repository we can get to has a small number of documents (43 if I remember correctly) and is at a low revision number (31 if thereabouts). The repositories that fail have many more documents and much higher revisions.

Also of note. Running Vault on Vista is extremely slow. I have resorted to running a Virtual PC with XP and Vault, on the same machine, using the same username that runs Vista and Vault, and the VPC is so much quicker for Vault to log in and display the repository list, and the VPC can access the Vault server and all repositories.

I have tried deleting the local cache, removed my working folders, reinstalling .NET 2.0 (ended up reinstalling the SDK as Vista doesn't like installing the redist due to being part ofthe OS). I've retrieved and viewed the vault server logs, which indicate that my login to Vault is working under Vista, and the data is retrieved successfully. I have not yet resorted to reloading the machine as I really don't want to.

Is there any help I can get on this, or somewhere I can look for the potential cause?

The details from the Vault Tech Support (Vault on Vista)
Client Information
Vault Client Version: 4.0.4.15848
.Net Framework Version: 2.0.50727.312
Operating System: Microsoft® Windows Vista™ Business
Service Pack: 0.0
OS Version: 6.0.6000
Total Physical Memory: 8 GB
Time Zone: (GMT+10:00) Canberra, Melbourne, Sydney

Server Information
Vault Server Version: 4.0.3.15836
.Net Framework Version: 2.0.50727.832
Operating System: Microsoft(R) Windows(R) Server 2003, Web Edition
Service Pack: 2.0
OS Version: 5.2.3790
Timezone: (GMT+12:00) Auckland, Wellington
SQL Version: Microsoft SQL Server 2005 - 9.00.3042.00 (X64)
Feb 10 2007 00:59:02
Copyright (c) 1988-2005 Microsoft Corporation
Standard Edition (64-bit) on Windows NT 5.2 (Build 3790: Service Pack 1)

License Information
27 serial number(s):
1 of 27: 1 full users, permanent
2 of 27: 1 full users, permanent
3 of 27: 1 full users, permanent
4 of 27: 1 full users, permanent
5 of 27: 1 full users, permanent
6 of 27: 1 full users, permanent
7 of 27: 1 full users, permanent
8 of 27: 1 full users, permanent
9 of 27: 1 full users, permanent
10 of 27: 1 full users, permanent
11 of 27: 1 full users, permanent
12 of 27: 1 full users, permanent
13 of 27: 1 full users, permanent
14 of 27: 1 full users, permanent
15 of 27: 1 full users, permanent
16 of 27: 1 full users, permanent
17 of 27: 1 full users, permanent
18 of 27: 1 full users, permanent
19 of 27: 1 full users, permanent
20 of 27: 1 full users, permanent
21 of 27: 1 full users, permanent
22 of 27: 1 full users, permanent
23 of 27: 1 full users, permanent
24 of 27: 1 full users, permanent
25 of 27: 1 full users, permanent
26 of 27: 1 full users, permanent
27 of 27: 1 full users, permanent

nickmoyle · Post by **nickmoyle** » Fri Aug 31, 2007 12:16 am

A further dig around and setting the Client logging, shows that the Get Repository is timing out.

Perhaps this large time delay difference I am getting between operation on XP vs Vista is the cause of the timeout.

Anyway excerpt from the client log is.

31/08/2007 4:11:35 PM <generic>: [GUIClientWorkerThread:4932] [System.Net.WebException: The operation has timed out.
at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.StreamReader.ReadBuffer(Char[] userBuffer, Int32 userOffset, Int32 desiredChars, Boolean& readToUserBuffer)
at System.IO.StreamReader.Read(Char[] buffer, Int32 index, Int32 count)
at System.Xml.XmlTextReaderImpl.ReadData()
at System.Xml.XmlTextReaderImpl.ParseEndElement()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at System.Xml.XmlReader.ReadString()
at System.Xml.XmlReader.ReadElementString()
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVaultService.Read21_VaultRepositoryDelta(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVaultService.Read173_GetRepositoryStructureResponse()
at Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer169.Deserialize(XmlSerializationReader reader)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)]The operation has timed out.
at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.StreamReader.ReadBuffer(Char[] userBuffer, Int32 userOffset, Int32 desiredChars, Boolean& readToUserBuffer)
at System.IO.StreamReader.Read(Char[] buffer, Int32 index, Int32 count)
at System.Xml.XmlTextReaderImpl.ReadData()
at System.Xml.XmlTextReaderImpl.ParseEndElement()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at System.Xml.XmlReader.ReadString()
at System.Xml.XmlReader.ReadElementString()
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVaultService.Read21_VaultRepositoryDelta(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVaultService.Read173_GetRepositoryStructureResponse()
at Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer169.Deserialize(XmlSerializationReader reader)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
[System.InvalidOperationException: There is an error in XML document (1, 16136). ---> System.Net.WebException: The operation has timed out.
at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.StreamReader.ReadBuffer(Char[] userBuffer, Int32 userOffset, Int32 desiredChars, Boolean& readToUserBuffer)
at System.IO.StreamReader.Read(Char[] buffer, Int32 index, Int32 count)
at System.Xml.XmlTextReaderImpl.ReadData()
at System.Xml.XmlTextReaderImpl.ParseEndElement()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at System.Xml.XmlReader.ReadString()
at System.Xml.XmlReader.ReadElementString()
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVaultService.Read21_VaultRepositoryDelta(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVaultService.Read173_GetRepositoryStructureResponse()
at Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer169.Deserialize(XmlSerializationReader reader)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
--- End of inner exception stack trace ---

Beth · Post by **Beth** » Fri Aug 31, 2007 9:53 am

When you are comparing your Vista performance to the VMware/XP performance, what version of VMware are you using?

Did you install the .msi as an administrator?

Here are the instructions we have for installing on Vista: http://support.sourcegear.com/viewtopic.php?t=7718

For the .NET portion, what I would recommend there is to open a command line, change directories to C:\Windows\Microsoft.NET\framework\2.0.......
and then run aspnet_regiis -u. When that is finished then run aspnet_regiis -i. That is the best way to perform an uninstall and reinstall of a .NET framework. Could you try it this way from the command line?

nickmoyle · Post by **nickmoyle** » Mon Sep 03, 2007 9:53 pm

Hi Beth,

Thanks for the information. I have tried as you suggested, and had no joy.

I have reinstalled a clean install of Vista Business 64bit, and Vault and still experience the same issue. I have not installed any other applications or drivers, only the Vista installation disk, and Vault. The error message is still the same "There is an error in XML document (1, 16137)." (note that the second number is always the same for this problem).

I log into Vault, after a long time (compared to running Vault on XP 32 bit) I get the repository list, and when selecting the main repository, I wait ten minutes to receive the error message.

When running under XP from the same computer, this process takes mere seconds (approx 15 at the most).

I have disabled the Windows Defender, UAC, Firewall, and malware protection. I have no other software other than Vista and Vault (I cannot stress this enough).

My coworker has a similar problem, but he uses Vista Home Premium 32bit. We have other co-workers located on the same site as the Vault server using Vista (not sure of versions) without this problem. We are located in a different country.

I can still access one of our repositories.
The repository I can access (after waiting a very long time) has the stats:
Rev 31, 45 files, 6 folders.

Examples of repositories that time out are:
Rev 50219, 100483 files, 7315 folders
Rev 5933, 2970 files, 336 folders
Rev 952, 737 files, 64 folders

Is there more information I can provide to you as this is very restrictive on our ability to work.

As previously stated I can run Vault on XP in a virtual pc (MS VPC2007) on the same machine that runs Vista, and it works correctly, without fail. Unfortunately, using a shared folder for a working folder takes a very long time when validation of files occurs (for Get Latest etc..), so I have had to create a batch file to xcopy my working folders from the VPC back to my normal working directory.

edit:

Wanted to add, the Vault server does not run under VMWare, the problem being experienced is only the Vault GUI client, and it only occuring on two users computers at this stage (as we are the only Vista users offsite).
Performance is not being scientifically measured when comparing Vista to XP, but then it doesn't need to. I've got some numbers for you by counting the seconds on my watch (so the times may be out by a second or two).

To get the repository list (8 repositories)
XP - 7 seconds
Vista - 19 seconds

To display the repository I can get to:
XP - 3 seconds
Vista - 823 seconds (no I am not kidding I can send you the log)

avonwyss · Post by **avonwyss** » Tue Sep 04, 2007 9:08 am

I'm not from SourceGear, but I have an idea that might help...

Are you working through an ADSL line (on either side)? Could it be that there is a MTU problem in your network? If yes, the problem could lie with the MTU packet size.

From Advances in Windows Vista TCP/IP http://blogs.msdn.com/wndp/archive/2006 ... pip-2.aspx:

Historically, problems due to the presence of black-hole routers have been among the highest product support call generators for the previous Windows networking stacks. To understand why, it’s important to know that TCP/IP relies on ICMP packet-too-big error messages to discover the maximum transmission unit (MTU) for any given connection’s path, so that it can reduce the size of the packets that it sends if they’re too large. If a router along the path does not send back ICMP error messages, or if a firewall drops ICMP error messages, TCP will never find out that its packets are too big. As a result, it will retransmit the packets repeatedly with the same size, up to its maximum number of retransmissions and, when it gets no responses, it will terminate the connection.

Black hole router detection is a mechanism used in this scenario to automatically reduce the size of the packets sent for a connection, based on the current status of the connection, in the absence of feedback from ICMP packet too big error messages. This mechanism was disabled by default in previous versions of Windows, because previous approaches would often yield too many false positives, lowering the packet size unnecessarily and reducing performance. In Windows Vista, our improvements have reduced the likelihood of false positives and, consequently, minimized the adverse performance impact, enabling us to turn on black hole detection by default in the upcoming Beta 2 release.

So, if for some reason this newly enabled feature does not work in your network setup, this could explain why XP has no problems and Vista does have problems. But it's just a thought...

Post by **jclausius** » Tue Sep 04, 2007 9:51 am

We haven't seen XML serialization / client hangs for the Max Transmission Unit problem, but it wouldn't hurt to check.

See Vault Client Can't Connect Through Some Routers (MTU) for more information about this topic.

avonwyss · Post by **avonwyss** » Tue Sep 04, 2007 9:58 am

The thing with the MTU is that it will only bite when the packet size grows too large. According to the problem reported, timeout and data problems seem to arise for larger transmissions.

Depending on the error handling, it may be that the data received so far is kept when the connection drops, and that this causes the XML deserialization error.

Post by **jclausius** » Tue Sep 04, 2007 10:19 am

You raise a valid point that the data will be variable based on what is being transferred. It won't hurt for the poster to have a look at any routers to see if MTU can be temporarily decreased in size to see if that has any impact on the problem.

Otherwise, we've seen things from bad .NET Framework installations to stateful firewall inspections or miscellaneous network devices interfering with the data in the network packets. In cases like this uninstalling / installing the .NET Framework might solve the problem. And, if that fails we've also used network monitoring tools such as Ethereal to track down the configuration problem.

nickmoyle · Post by **nickmoyle** » Wed Sep 05, 2007 2:28 am

All good ideas. I am running over ADSL to the big wide world, and have been for over 3 years now, using many versions of Vault. All have worked successfully in the past.

I have tried disabling the PMTU to get the same behaviour as XP's TCP/IP stack, although I cant see how black hole discovery would interfere. It should improve the connection.

I have also validated the MTU settings. My local router was set to 1300. I've moved this down to 1024, and up to 1492 (the max it supports). Neither have had any effect (the other end is more difficult to change). I fail to see how this would change much as my XP setup (both VPC and a seperate laptop) both connect fine from behind the same router as the Vista boxes, which do connect but fail to load the repository information.

We have also tried other locations using different ISP's and different routers. We cant work out why the connection between two sites would work for one OS and not another?.

Also trying to get a network capture to determine if the full XML document is arriving and its just not being processed (according to the server logs it is being successfully sent).

avonwyss · Post by **avonwyss** » Wed Sep 05, 2007 2:45 am

That's indeed quite strange. Since it works with XP and not with Vista, my guess was some sort of network setting. Is there maybe a setting in the Vista TCP/IP stack to drop fragmented packets (this would be a security feature to avoid DoS attacks by packets marked fragmented)? Assuming that the packets get fragmented on the server side (Ethernet MTU is 1500, ADSL max is 1492), this would lead to different behvior on XP and VIsta. But since I don't have Vista I cannot try for myself and I'm just guessing...

Something that cannot hurt in any case is to set the MTU of the Vault server and of Vista as well as the one of your router to a common value (I'd start with 1492 and go down to 1300 or so) to rule out fragmentation issues altogether. In the worst case, this will add a tiny bit of overhead since the header-to-data ratio of the packets is lower and the number of packets a little bit higher, but on the other side, if it does avoid fragmentation, it can also enhance performance.

Again, I'm not affiliated with SourceGear and they may have other ideas to solve your problem.

nickmoyle · Post by **nickmoyle** » Wed Sep 05, 2007 8:11 pm

All help is good help

. I have been running network stack traces, and the standard packet size I'm receiving from the Vault server is 1452, so with any overhead this should fit within the packet frame.

I have been connecting to this same Vault server via XP with a MTU of 1300 for over a year now, so I suspect fragmentation isn't the issue.

From the network traces, it looks like Vista isn't always sending an ACK on receive, although I don't understand why (as this should be a stack comm the problem is not likely to be Vault unless its not clearing the buffer??), so I'm disabling bindings left and right, and hoping that I'll stumble across the reason soon. Will try another port soon, and another NIC, and any other ideas I or anyone else can come up with.

nickmoyle · Post by **nickmoyle** » Thu Sep 06, 2007 2:13 am

I now have lots of bad things to say about Microsoft and their so called new and wonderful features that are not implemented correctly.

After running traces in both XP and Vista, I noticed the Window size from Vista was extremely low. Around 250 bytes - 260 bytes.

So after a bit of searching I came across this gem on KB http://support.microsoft.com/kb/935400

And after following the instructions and disabling this wonderful Window Scaling feature, Vault works great on Vista and there is pretty much 0 difference in reponse between Vista and XP.

Thanks to those who posted their thoughts.

avonwyss · Post by **avonwyss** » Thu Sep 06, 2007 2:28 am

Thanks for the feedback. So it wasn't the black hole

detection, but rather the scaling autotuning... those autotunig features seem to be wrong quite often.

nickmoyle · Post by **nickmoyle** » Thu Sep 06, 2007 2:43 am

This article http://www.microsoft.com/technet/commun ... g1105.mspx has an interesting comment in the middle

Note Some Internet gateway devices and firewalls block packet flows because they do not correctly interpret the scaling factor used in TCP connections. Because of this, Internet Explorer in Windows Vista uses an initial scaling factor of 2. Other applications use a default initial scaling factor of 8. Microsoft is investigating changing the initial scaling factor for Internet Explorer-based connections to 8 in a future update of Windows Vista. Microsoft is working with the manufacturers of these devices so that they can be updated for compliance with TCP window scaling.

So I guess I must have one of these gateway/firewalls that doesn't do the right interpretation between me and my Vault server.

Post by **jclausius** » Thu Sep 06, 2007 8:48 am

I'm glad you were able to resolve this problem.

I haven't read about any of the so-called new "performance optimizations" for Vista. So these things always come as a bit of a surprise. In any case, I'm going to experiment with the auto Window Scaling factor as I've had poor network performance on my Vista machine over 1Gb networks.

Thanks for the post back.