xbox-scene.com archived forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

xbox-scene.com forum restored.  registration disabled.  thanks to xboxexpert for the db dump and netham45 for cleaning it up!

Pages: [1] 2 3 ... 5

Author Topic: X360 Backwards Compatibility  (Read 910 times)

PedrosPad

  • Guest
X360 Backwards Compatibility
« on: March 17, 2006, 07:17:58 AM »


I'm using this thread to pull together related info from disparate threads in this and other forums.  I hope no one minds, but I think the whole may be more than the sum of the parts.

Overview:
The XBOX 360 console can run games from the M$ console that proceeded it, the XBOX.  This is known as  backwards compatibility.  Due to major hardware differences between the two consoles, on the X360 this is achieved by software emulation of the legacy XBOX.  Software emulators 'interpret' the legacy program's code and ensure the host produces the same effect.

Quote from M$'s article "Q & A: Backward Compatibility":
QUOTE
An early version of the emulator that supports Halo�: Combat Evolved and Halo� 2 offline is included on Xbox 360 Hard Drives right out of the box as a special bonus to devoted fans of the franchise. However, to play Halo 2 online, or to play any other titles on the launch list, the full emulator update is required.

This quote informs us that
  • 1) M$'s XBOX software emulator is incomplete.
  • 2) M$'s XBOX software emulator is updateable.
Shipping incomplete mandated that the emulator needed to be updatable, but why is it incomplete?  Emulation of another platform (i.e. XBOX), is very challenging.  Recall that the emulators purpose is to interpret and 'synthesize' the user experience the customer would have enjoyed on the original XBOX.  The quantity and diversity of XBOX titles each make different demands of the emulator.  Thus the sensible approach is develop the emulator incrementally; building in support for more and more titles as each new 'demand' is encountered.

Of course, multiple titles may make identical demands, so once one is supported, the remainder will be also.  But don't be mistaken into thinking that titles from a genre, such as RPG, or FPS, would make common demands.  In fact the reverse is true.  Market forces dictate that titles published into a crowded genre need to outshine their competition.  This is typically achieved by being technically diverse (a.k.a. "pushing the limits").

Better estimation and categorisation of the emulation demands of a title can be made based on the XDK with which it was developed.  Subsequent XDK releases introduced new, richer, functionality.  Thus titles that were developed in the same time period are more likely to make common demands.

A second quote from the article quoted above:
QUOTE
As we focused on the top sellers, we discovered that many other games worked due to similarities in their technology. As an added benefit, games that share engines and technology have been in some cases easier to certify for backward compatibility and have made the launch lineup as a result.


Background:
Most software today is written using calls to libraries of existing code, sequenced by a primary process.  These libraries contain pre-written and tested code to perform common activities (such as playing WAV sound files, or draw shapes on the screen).  Use of such libraries makes software development rapid.  The M$ XDK provided a wealth of such libraries, allowing title developers to focus on their "primary process" - their unique game!

Once support for specific a library is added to the emulator, all the titles using that specific library move one stage nearer being supported.  Once all the libraries a title relies upon are supported, the title has a good chance of working.

Technical:
The Windows utility deXBE can open an XBOX executable XBE file, and reveal the libraries it uses.
IPB Image

It's easy to imagine M$ writing a similar utility to scan all the XBEs for details of the XDK libraries utilized and built up a 'title library requirements database'.  As soon as they're confident a specific XDK release of a library is supported by BC, any titles whose 'total library demands' are now met are cleared for retest on X360.  Although most titles would have a very good chance of working fine, some may still fail for a variety of technical reasons, and would be pulled for targeted diagnosis.  Others 'appear' to be working and pass the inspection, but are later found to have issues.  As is evident here and here (quoted below):
QUOTE
The original list was composed of 213 games and was unveiled on November 11, 2005 on the Xbox.com website; however, games have been subsequently removed due to bugs. The following 13 games have been removed from the list: Catwoman, Codename: Kids Next Door, Curse: The Eye of Isis, NBA LIVE 2003, Sid Meier's Pirates, Star Wars: Clone Wars, Blinx 2, FIFA World Cup 2002, Grabbed by the Ghoulies, Legends of Wrestling, Mortal Kombat: Deception, and Rugby 2005.


Although checking if an XBE's libraries are supported by BC could happen on the X360 console itself, it's more practical (and provides greater control) if all this work happens back at M$, and the X360 BC updates simply amend a list of supported XBE title IDs in the X360's backwards compatibility database (believed to be the TDBX\Tdbx.db file on the X360 HDD).  (At >150MBs the TDBX\Tdbx.db file obviously contains more than just the XBE title IDs!  Given that length, it probably contains necessary patches/detours to specific XBE libraries that are applied between the emulator loading the XBE, and it beginning execution.)

That said, some titles not on the official BC list are known to work, most notably, the menu systems on the OXM demo disks.  Gathering library information for the titles the emulator currently supports could be an interesting exercise, and provide hints toward additional titles that may also work.

Future:
Deconstructing the TDBX\Tdbx.db backwards compatibility database file could be a revealing exercise.  It may list additional titles that could be played.  Eventually understanding of this file may also facilitate the injection of new titles - permitting all sorts of mischief cool.gif.

A possible step toward interpreting this file would be to incrementally install each of the downloadable BC updates, and use a hex editor to examine the file following each update.

This is just what I am planning to do - as and when time allows.

In order to avoid a false start, I'd like to ask a few questions here.

Do the X360 Backwards Compatibility updates, when installed from CD, update the firmware at all, or is their 'effect' limited to the HDD?  (Read on for how to determine.)

My X360 is as stock and has firmware 2.0.1888.0.  (To view this information simply go to your dashboard. Then go to the system tab. Next go to console settings. Then go into system info.)

Has anyone else not gone on XBOX!Live yet, and only used the downloadable BC updates?  If so, what is your firmware version?  Known versions are:
QUOTE
  • The Xbox 360 ships with kernel and dashboard 2.0.1888.0
  • At the product launch an update was available that updated the kernel and dashboard to version 2.0.2241.0.
  • An update released on January 30 2006 updates the kernel and dashboard to version 2.0.2255.0.


For completeness - IIRC There's been 2 downloadable X360 BC updates so far.
  • November 11, 2005 - supporting 213 XBOX1 games.
  • December 9, 2005 - Six Tom Clancy games were added and Freedom Fighters was removed due to glitches.
These should provide 3 versions of the TDBX\Tdbx.db backwards compatibility database file:
  • The stock Tdbx.db file shipped on the HDD (HALO 1/2 emu only?)
  • A Tdbx.db file containing the November updates
  • And, a Tdbx.db file containing the December updates.





Edited by PedrosPad, 17 March 2006 - 08:39 PM.


Logged

hitsndc202

  • Guest
X360 Backwards Compatibility
« Reply #1 on: March 17, 2006, 09:26:44 AM »


"Do the X360 Backwards Compatibility updates, when installed from CD, update the firmware at all, or is their 'effect' limited to the HDD? (Read on for how to determine.)"

That only goes to the HDD, no need for the firmware at all. Hence that is why u need the HDD to play old games. wink.gif   U have a good thing stiring up,  I like it! Hope this info helps u out man.




Logged

PedrosPad

  • Guest
X360 Backwards Compatibility
« Reply #2 on: March 17, 2006, 11:27:08 AM »


"Do the X360 Backwards Compatibility updates, when installed from CD, update the firmware at all, or is their 'effect' limited to the HDD? (Read on for how to determine.)"

That only goes to the HDD, no need for the firmware at all. Hence that is why u need the HDD to play old games. wink.gif

Thx for the feedback  smile.gif .

I'm really looking for someone who is running the latest BC updates and can confirm that they're still running X360 firmware 2.0.1888.0.  (Wow.  That sure took less words to ask - lol  laugh.gif )

At >150MBs the BC DB is too large to practically store in the Flash ROM (which is why it must live on the HDD), but it could be 'secured' by a hash/checksum that is stored in the Flash ROM!




Edited by PedrosPad, 17 March 2006 - 08:42 PM.


Logged

l33tphreak

  • Guest
X360 Backwards Compatibility
« Reply #3 on: March 18, 2006, 10:51:10 AM »


actually i just got the latest update today because i tried to go on live and was given the error "this console needs to be updated before it can be signed into live. without this update you will not be able to sign into live." (or something to that effect, i dont remember it verbatim) so anyway my current dashboard version is 2.0.2258.0 i hope this helps.




Logged

PedrosPad

  • Guest
X360 Backwards Compatibility
« Reply #4 on: March 20, 2006, 04:58:03 AM »


Ok.  I spent the weekend conducting the tests indicated in the root post.  Lots of new info, and ideas dismissed.

Results:
To establish a baseline, I attempted to boot my BC test title, Fable, on the Virgin X360.  It failed with a message saying go and download the BC updates.  This had been seen before and was fully expected.

First off, the first BC update of Nov05, does update your firmware (Dashboard and Kernel).  Booting this from DVD-R took my firmware from �D:2.0.1888.0� to �D:2.0.2241.0 K:2.0.2241.0 (BK: D:2.0.1888.0)� (So there�s no going back - eek! sad.gif ).  Another surprise is that comparing the HDD contents between the Virgin HDD, and the one with Nov05 BC update installed showed it didn�t change any files.  It did add two files, 2 PIRS files in 2 new folders \$systemupdate\ & \$titleupdate\.  The TDBX\Tdbx.db wasn�t changed at all!  My BC test title, Fable, booted fine.

Installing the Dec05 BC update subsequently didn�t change the TDBX\Tdbx.db either.  In fact, this time, the only file that was changed was the PIRS file in \$titleupdate.  The Dec05 BC update didn�t appear to update the dashboard or kernel further (it still reports �D:2.0.2241.0 K:2.0.2241.0 (BK: D:2.0.1888.0)�).  My BC test title, Fable, still booted fine.

All this left me scratching me head..... huh.gif

Further tests:
I�d read that people who have updated from XBL don�t have the \$systemupdate\ & \$titleupdate\ folders, thus these can�t be fundamental to the BCs operation.  Given that these were the only visible changes on my HDD filesystem, I theorised that the BC functionality and updates must all be stored in the Flash firmware.  To test this theory, I reverted the HDD back to it's virgin state (which would have simply removed the $* folders) (obviously I'm not able to revert the firmware sad.gif) and booted Fable expecting it to work � it didn�t! and recommended that I install a BC update.  huh.gif  Puzzling. uhh.gif  I recalled seeing on the free60 site that the X360 HDD gains a new partition at some stage (see here), so I now suspect that this may be what it was looking for.  Further tests in this direction shortly�

(Another side finding was when updated to �D:2.0.2241.0 K:2.0.2241.0 (BK: D:2.0.1888.0)� via the Nov05 BC update, the "Experience 360" Kiosk demo disk continued to work.   smile.gif )




Edited by PedrosPad, 20 March 2006 - 04:22 PM.


Logged

Angerwound

  • Guest
X360 Backwards Compatibility
« Reply #5 on: March 20, 2006, 05:33:16 AM »


Alongside Pedro, I've been conducting a few tests of my own to hopefully disprove a few of his theories!  tongue.gif

My first test being the removal of the 'TDBX.db' file that was suspected to house Backward Compatibility information. Removing the file and executing a few legacy titles resulted in no problems. Each game booted and ran just fine. The two used were Halo 2 and Medal of Honor: Rising Sun.

So I began to become more interested in what exactly is happening when legacy titles are executed. The only other pieces that could be getting used are the 'xbox.XEX' and 'xefu.XEX'.

My first test was to remove both of the files to be sure they in fact were getting executed. Once removed, a legacy title would result in the error, 'Disc is Unreadable or Scratched' and a return to the 360 Dash.

Secondly, I removed only the 'xefu.XEX' executable. Again, boot of a title would result in the 'Unreadable' error mentioned above but it however did boot the 'grey xbox boot screen' right before. So this leads me to believe that the 'xbox.XEX' is being executed first.

Next, i left 'xefu.XEX' and removed 'xbox.XEX'. This resulted in the first test results; no grey xbox screen and the unreadable disc error.

Since I now know that the 'xbox.XEX' is being executed first. I figured why not switch the filenames and trick the 360 into booting the 'xefu.XEX' first. This simply didn't work and resulted again in no grey screen and the Disc is Unreadable error.

My findings lead me to believe that all backward compatibility is done via the flash chip containing the dashboard or the kernel somewhere. The latter seems less likely with Pedro's tests mentioned above. (DEC Update doesn't update the kernel).

Take this info and apply it to your own research, maybe we'll finally locate that infamous BC data!




Edited by Angerwound, 20 March 2006 - 02:34 PM.


Logged

jizmo

  • Guest
X360 Backwards Compatibility
« Reply #6 on: March 20, 2006, 05:34:45 AM »


Good work on the subject, PedrosPad. I don't exactly know what to make out of the results, changes in Tdbx.db would've been clearly the easiest to track down - obviously it's going to be much more difficult than that. sad.gif

DVD media check hack - if it's really working - certainly brings some interesting aspects in studying BC as well. On top of modifying data files there are also certain things you can change in a XBE and still run it. It'd be interesting to see if the emulator notices changes in title, icon etc other data, or does it refuse the run it afterwards.

Who knows, maybe they've had relied on that media check and title data would be sufficient means of securing the running of wanted XBE's, which would leave an exploitable gap in the BC system.

And yes, I know, it's a long shot biggrin.gif

EDIT: Props to Angerwound as well for the additional info




Edited by jizmo, 20 March 2006 - 02:39 PM.


Logged

PedrosPad

  • Guest
X360 Backwards Compatibility
« Reply #7 on: March 20, 2006, 05:58:44 AM »


My findings lead me to believe that all backward compatibility is done via the flash chip containing the dashboard or the kernel somewhere. The latter seems less likely with Pedro's tests mentioned above. (DEC Update doesn't update the kernel).

Since the Nov05 BC update is no longer downloadable from M$, new users would only be able to get the current Dec05 BC update.  But since they could/would still be running "D:2.0.1888.0", this demands that that Dec05 update also include the firmware update to �D:2.0.2241.0 K:2.0.2241.0�.  But because I was already running �D:2.0.2241.0 K:2.0.2241.0 (BK: D:2.0.1888.0)�, I wouldn't have seen any change.

A question is, does the Dec 05 BC update make changes to the firmware, but not update the Dash/Kernel version numbers?  Maybe the BC functionality isn't considered to be part of the Dash/Kernel!. unsure.gif

The X360 HDD is certainly needed for BC, but maybe it's only to needed in place of the XBOX1 HDD (i.e. storage space for HALO to cache its maps, etc.).  Maybe it serves no other role in BC.  unsure.gif

Edit:
December 9, 2005 - Six Tom Clancy games were added and Freedom Fighters was removed due to glitches.
Ok, so if I restore the X360 HDD to the state after the Nov05 BC update (so any new BC HDD partitions exist, but no Dec05 files!) (I now have a library of HDD images!  happy.gif ), then attempt to run Freedom Fighters, if it doesn't work it means that the Dec05 BC update has updated my Flash firmwareIPB Image  Heads off to rental store....




Edited by PedrosPad, 20 March 2006 - 03:35 PM.


Logged

PedrosPad

  • Guest
X360 Backwards Compatibility
« Reply #8 on: March 20, 2006, 07:25:12 AM »


My first test being the removal of the 'TDBX.db' file that was suspected to house Backward Compatibility information. Removing the file and executing a few legacy titles resulted in no problems. Each game booted and ran just fine. The two used were Halo 2 and Medal of Honor: Rising Sun.

Any chance this was operating from data in the cache?  I'm wondering if this TDBX.db data file may be used in the dynamic recompilation of a BC title, which then gets stored in the cache uhh.gif

If it's not used for BC, what the heck is the TDBX.db file used for? and why is it in the X360's \Compatibility\ folder?  huh.gif




Edited by PedrosPad, 20 March 2006 - 06:19 PM.


Logged

PedrosPad

  • Guest
X360 Backwards Compatibility
« Reply #9 on: March 20, 2006, 10:01:14 AM »


I recalled seeing on the free60 site that the X360 HDD gains a new partition at some stage (see here), so I now suspect that this may be what it was looking for.  Further tests in this direction shortly�

Ok, I restored the Virgin X360 HDD image, and then installed the Dec 05 BC update from DVD-R.  I then diffed the Virgin & updated HDD images.  The only differences in the 20GB HDD images are additional directory entries for the two new folders ( \$systemupdate\ & \$titleupdate\), entries for the 2 files they contain, and the file contents themselves.  Nuffin else.  huh.gif I had 4 partitions on the Virgin HDD, and same four on the updated HDD.

But when I try to play BC title Fable with the Virgin HDD, I'm requested to install BC updates. blink.gif My best guess is that it's the contents of the \$titleupdate folder it's looking for.  Are you XBL guys who play BC titles sure you don't have an \$titleupdate folder?

IPB Image




Edited by PedrosPad, 21 March 2006 - 10:12 AM.


Logged

Angerwound

  • Guest
X360 Backwards Compatibility
« Reply #10 on: March 20, 2006, 10:49:40 AM »


Positive, I've got no $systemupdate or $titleupdate directories what so ever on my hdd. All my updates have been done via the live service.




Logged

DaBiscuit

  • Guest
X360 Backwards Compatibility
« Reply #11 on: March 20, 2006, 11:44:38 AM »


Positive, I've got no $systemupdate or $titleupdate directories what so ever on my hdd. All my updates have been done via the live service.


I'd guess the next step in the process is to discover if there are any other differences between an XBL-updated HDD and a manually-updated HDD. This whole thing doesn't make sense otherwise.




Logged

PedrosPad

  • Guest
X360 Backwards Compatibility
« Reply #12 on: March 21, 2006, 02:41:36 PM »


Ok, I plan to revisit and redo my X360 HDD BC tests in the near future.  For now I turned my attention to the PIRS archives within the $systemupdate & $titleupdate folders.

The contents of the PIRS archive within the $systemupdate folder is identical between the Nov 05 BC update, and the Dec 05 BC update.  I strongly suspect this is the Dash & Kernel update to 2.0.2241.0.  It contains 23 files, most with an *.XEXP file extension.  *.XEXP files tend to be quite small (compared to XEXs of the same name).  I suspect these are some sort of XEX patch files - akin to IPS patch files.  The contents of the *.XEXP files also support this theory when viewed using a hex editor � short records of varying lengths.
IPB Image

The PIRS archive in the $titleupdate folder is different between the two BC updates.  The Nov 05 BC update PIRS archive contains:
xbox.xexp
xefu.xexp

And the Dec 05 BC copy contains:
IPB Image

On the X360 HDD, within the \Partition 2\Compatibility\ folder, there exists xbox.xex and xefu.xex � the apparent targets for the *.XEXP patch files!  However these files never change from the original copy in the Virgin HDD image, even after the final Dec 05 updated is installed!  Maybe these �patch files� are merged in dynamically, as the base file is loaded?  If the firmware updates work the same way, it may explain the BK: bit of the Dash/Kernel version number �D:2.0.2241.0 K:2.0.2241.0 (BK: D:2.0.1888.0)� (although the *.XEXP patch files themselves must be written into the Flash, as the Dash/Kernel remains at version 2.0.2241.0 when you boot without the HDD connected!).

Finally, notice the xefu2.xex file.  This filename doesn�t already exist on the HDD, and thus this must be supplied complete, as there is no existing target for a xefu2.xexp patch files to merge with!

(In a bid to head off the obvious question - I�m sure the byte changes in the patch files also patch any signatures as necessary � preventing them being used to apply any non-authorised patches!)




Edited by PedrosPad, 22 March 2006 - 01:15 PM.


Logged

PedrosPad

  • Guest
X360 Backwards Compatibility
« Reply #13 on: March 21, 2006, 03:48:59 PM »


Often the actual changes to the files are minimal.  By using dynamically loaded 'patch files', M$ is able to leave the entire base 2.0.1888.0 OS in the Flash untouched, and only needs to burn these small patch files into subsequent Flash memory areas.  Preserving the rewrite limit for when it might be needed.

Even more thoughts on this.... wink.gif
Apart from the blatant efficiency in download band width, a security advantage of distributing OS updates as �Fusion� � patch files has occurred to me.

Due to the X360 being a 'closed system', M$ knows the finite range of possible installed foundation OSes.  This fact permits them to author and distribute OS updates as simple patch files - bring each level up to current.  This update model also provides additional security, since anyone intercepting the update only ends up with the patch file � which is meaningless without the foundation!

Should the patch file format consist of file offsets and replacement bytes, successive updates could be applied to a blank file, in an attempt to build up the complete file.  However, if the patch file format is more along the lines of �Take foundation file bytes 50-to-56 and move them to 100-106, peek foundation file byte offset 25, add 12, then poke into offset 160�, then no amount of patch files will get you nearer the complete file.  (A modern instance of a �book code�!  The X360 knows the contents of the Flash memory, and M$ knows the contents of the Flash memory, but we don't!)  Clever eh?

It�s too early to tell if this is what M$ have done at this point.  But we should be able to find out by examining the xbox.xexp patch file and the complete xbox.xex file we do have from the HDD, and see if interpreting it as �offset/replacement byte� style format results in a sensible file.

(Of course nothing limits M$ to using patch files.  I'm sure they'll only use them for minor updates, and only for as long as it suits them.  A comprehensive Flash update may still be delivered as a big encrypted binary file at some future point.)




Edited by PedrosPad, 22 March 2006 - 12:13 PM.


Logged

'PedrosPad'

  • Guest
NULL
« Reply #14 on: March 22, 2006, 02:31:26 AM »

'
               
QUOTE(PedrosPad @ Mar 22 2006, 01:55 AM) View Post

Often the actual changes to the files are minimal.  By using dynamically loaded 'patch files', M$ is able to leave the entire base 2.0.1888.0 OS in the Flash untouched, and only needs to burn these small patch files into subsequent Flash memory areas.  Preserving the rewrite limit for when it might be needed.

Even more thoughts on this.... wink.gif
Apart from the blatant efficiency in download band width, a security advantage of distributing OS updates as �Fusion� � patch files has occurred to me.

Due to the X360 being a 'closed system', M$ knows the finite range of possible installed foundation OSes.  This fact permits them to author and distribute OS updates as simple patch files - bring each level up to current.  This update model also provides additional security, since anyone intercepting the update only ends up with the patch file � which is meaningless without the foundation!

Should the patch file format consist of file offsets and replacement bytes, successive updates could be applied to a blank file, in an attempt to build up the complete file.  However, if the patch file format is more along the lines of �Take foundation file bytes 50-to-56 and move them to 100-106, peek foundation file byte offset 25, add 12, then poke into offset 160�, then no amount of patch files will get you nearer the complete file.  (A modern instance of a �book code�!  The X360 knows the contents of the Flash memory, and M$ knows the contents of the Flash memory, but we don't!)  Clever eh?

It�s too early to tell if this is what M$ have done at this point.  But we should be able to find out by examining the xbox.xexp patch file and the complete xbox.xex file we do have from the HDD, and see if interpreting it as �offset/replacement byte� style format results in a sensible file.

(Of course nothing limits M$ to using patch files.  I'm sure they'll only use them for minor updates, and only for as long as it suits them.  A comprehensive Flash update may still be delivered as a big encrypted binary file at some future point.)
               
               

               


                     Edited by PedrosPad, 22 March 2006 - 12:13 PM.
                     
                  


            '
Logged
Pages: [1] 2 3 ... 5
 

Page created in 0.123 seconds with 15 queries.