sdk RFC: dotnet [Arm64 | x64] coexistence - Csharp

Microsoft and Apple are both producing new OSes for Arm64 that support x64 emulation. For .NET, we need to determine how to support .NET Arm64 and x64 builds on those OSes in a way that offers a reasonable user experience and that has reasonable cost.

Problem to solve

As stated, we need to offer x64 and Arm64 builds of .NET within one 64-bit namespace (without much help from the OS on how to manage it). That leads to three issues:

  • How do we separate the products in some directory structure (peer directories or unified)?
  • Do customers manage using the arm64 vs arm64 product via which one is in the path (or absolute paths to dotnet)?
  • Do the directory names represent the product in the way we want?

One could reasonable ask why we need to support coexistence once native Arm64 build are available.

  • We are only building .NET 6 for Apple Arm64, and people will have legitimate need to use earlier .NET versions that are x64-only.
  • The situation on Windows is more favorable since both .NET 5 and 6 are built for Windows Arm64, but the general problem is the same.
  • The point of emulation is to enable a smooth transition to Arm64. It may take some time for people to transition to Arm64. For example, someone may be able to move to .NET 6 immediately but not to Arm64 because of a native dependency.
  • If we decided not to solve this problem, we'd want to consider blocking the x64 installers on macOS and Windows Arm64 machines, or at least showing a warning that there are unsolved problems that the user will likely run into.

Context

The Windows and macOS plans and implementations are similar but have key differences, per our understanding.

- Same: No WoW64 subsystem or Program Files (x64) style experience. For example, there is no x64 command prompt.
- Same: An executable can be universal.
- Different: macOS universal executables are transparently restartable, while Windows ones are not.
- Different: Windows x64 emulation is here to stay, while macOS x64 emulation will probably be removed in 3-5 years.

The following are various high-level solutions that we could employ, with pros and cons.

Status quo

On Windows x64, .NET is installed to two locations (depending on architecture):

- C:\Program Files\dotnet
- C:\Program Files (x86)\dotnet

Today, customers need one of those two directories in the path in order to get the "dotnet" experience they want, like if they want a 32-bit or 64-bit "dotnet build".

Note: There will not be a C:\Program Files (x64) directory on Windows Arm64. We are expected to install 64-bit products (Intel and/or Arm based) in C:\Program Files.

On macOS, .NET is installed to one location:

- /usr/local/share/dotnet

Going forward

There will be one dotnet in the path, just like today. Much like the 32- and 64-bit support we offer on Windows, we'll offer both Arm64 and x64 builds of .NET and customers can install both or either, and can control which is in the PATH. That will determine if they get an Arm64 or x64 dotnet build. We are not planning on building a .NET version manager that enables switching which architecture you get. We intend to rely on the PATH.

The question is what structure we offer for the two 64-bit products, how intuitive that is (now and later) and how expensive that is for us to offer.

Native dotnet

Premise: there is a "dotnet" directory on every OS, and it is the native architecture.

We'd end up with the following, on Windows and macOS, respectively:

  • C:\Program Files\dotnet
  • C:\Program Files\dotnet_x64
  • /usr/local/share/dotnet
  • /usr/local/share/dotnet_x64

This is the usual "who gets the good name" problem.

Pros: - The dotnet directory is the native architecture on all Oses. - This option is particularly beneficial for macOS, since we expect x64 emulation to be taken away relatively quickly, at which point dotnet_x64 would disappear on that OS.

Cons: - x64 directories on x64 and Arm64 machines no longer match. - x64 installers needs to change, including for 3.1 and 5.0. Hopefully, a single install can install to dotnet on an x64 machine and dotnet_x64 on an Arm64 machine. If not, we need a second set of x64 installers. That's likely untenable. - We have to break, migrate or somehow manage existing x64 .NET installs on Apple Silicon machines.

Archify dotnet

Premise: fully embrace multi-arch support, with arch-specific directories.

We'd end up with the following:

  • C:\Program Files\dotnet_arm64
  • C:\Program Files\dotnet_x64
  • /usr/local/share/dotnet _arm64
  • /usr/local/share/dotnet _x64

On Arm64, we'd have these arch-specific directories. This is the "no one gets the good name option; everybody loses" option.

Pros: - The directories are all self-descriptive and symmetrical.

Cons - Same cons as "Native dotnet" - Install directories on Windows x64 and Windows Arm64 don't match. - We're stuck with dotnet_arm64 on macOS forever, even though that's the only architecture supported (in the long run).

Hide architecture differences

Premise: These differences don't need to be so apparent. We already have version folders under dotnet. We can add arch folders (or something similar). This option has a lot of sub-options, too.

Option 1 -- Insert a new folder, with discrete .NET hives underneath:

  • C:\Program Files\dotnet\arm64
  • C:\Program Files\dotnet\x64
  • /usr/local/share/dotnet/arm64
  • /usr/local/share/dotnet/x64

Option 2 -- Intermix architectures in one structure (here, just shown with Windows, for simplicity):

  • For "dotnet"
    • C:\Program Files\dotnet\arm64\dotnet.exe
    • C:\Program Files\dotnet\x64\dotnet.exe
  • For frameworks
    • C:\Program Files\dotnet\sharedarm64\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\
    • C:\Program Files\dotnet\sharedx64\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\

Put "x64" and "arm64" somewhere in the folder hierarchy. Where it is, is an implementation decision. It is the same (in spirit) as C:\Windows\Microsoft.NET\Framework and C:\Windows\Microsoft.NET\Framework64.

Option 3 -- Arm64 is native architecture with x64 intermixed in:

  • For "dotnet"
    • C:\Program Files\dotnet/dotnet
    • C:\Program Files\dotnet/x64/dotnet
  • For frameworks
    • C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\
    • C:\Program Files\dotnet\shared_x64\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\

Pros: - We can manage Arm64 and x64 builds within a single structure in a way that we see as best, both now and over time.

Cons - Same cons as "Native dotnet"

Hide architectural differences with universal binaries

Both Windows and macOS offer a form of universal binaries, which we could use as a multiplexer between the two 64bit products. There are two problems with that.

  • We don't have enough information in all scenarios to know which architecture is the right one.
  • The Windows implementation (per our understanding) isn't as feature rich as we'd need to make all scenarios work, like debuggers.

We're rejecting this option for now. We could decide to adopt this solution on macOS and not Windows, should feedback push us in that direction.

Compatibility Wins

Premise: People have already installed .NET x64 on Apple Silicon machines. We need to respect that. This is the opposite of "Native dotnet"

We'd end up with the following:

  • C:\Program Files\dotnet_arm64
  • C:\Program Files\dotnet
  • /usr/local/share/dotnet _arm64
  • /usr/local/share/dotnet

Pros: - Good compatibility for existing Apple Silicon .NET users.

Cons: - Windows doesn't necessarily need this solution. We can change .NET before it ships with x64 support. - x64 gets the "good name" forever.

Hard design questions

  • Developer UX
    • Would it be good enough to only have the platform native dotnet available on PATH by default (Native Dotnet solution above)?
    • On Windows x64 the native .NET is typically the one on the PATH is already the case since typically only the x64 dotnet is on PATH, but it's possible to use it to build, run and test x86 apps.
  • User UX
    • can we fully rely on apphost to solve the problem of picking the right archtiecture to run the app with? Different way to ask this: How common is it to run applications via "dotnet app.dll"?
    • Is it possible to release new versions of downlevel installers (3.1 and 5) for macOS x64 and Windows x64 which would be ARM64 aware and could move the default location of the downlevel runtimes?
    • How hard is it to create an x64 installer which can detect that it runs on ARM64 hardware and can change behavior based on that? This is dependent on the installer technology on Windows and macOS.

Other considerations

Asked Oct 14 '21 08:10
avatar richlander
richlander

9 Answer:

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

1
Answered Apr 14 '21 at 02:56
avatar  of dotnet-issue-labeler[bot]
dotnet-issue-labeler[bot]

This issue is a kind of RFC.

I'm currently leaning to "Hide architectural differences" option 3. It offers much of the same experience as "Native dotnet" while providing a little more wiggle room than some of the other options.

1
Answered Apr 14 '21 at 02:59
avatar  of richlander
richlander

"Hide architectural differences" option 3

+1

C:\Program Files\dotnet\shared_x64\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\

Why not park the frameworks, sdks and everything else under the x64 directory? ie make this C:\Program Files\dotnet\x64\shared\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\, C:\Program Files\dotnet\x64\sdk\..., etc. The advantage of doing it this way is that everything in x64 environment can assume the same directory structure relative to dotnet.exe.

1
Answered Apr 14 '21 at 03:20
avatar  of jkotas
jkotas

That makes total sense. My main point was that the x64 can go anywhere, but your proposal seems to be the best one for the reasons you've shared. Nice.

I'm not going to update the document now. Let's wait for a first round of review and then produce a new doc with the final plan.

1
Answered Apr 14 '21 at 03:21
avatar  of richlander
richlander

I really like Hide Architectural Differences option 3.

Arm64 native with x64 mixed in, keeps everything together with a less confusing folder structure and would likely solve the issue of the hostfxr.dll being loaded for the incorrect architecture due to the incorrect x64 installation path - one of the big pains with arm64 and x64 coexistence.

I would absolutely LOVE to never see this again:

Message: Failed to load the dll from [C:\Program Files\dotnet\host\fxr\6.0.0-preview.3.21201.4\hostfxr.dll], HRESULT: 0x800700C1
The library hostfxr.dll was found, but loading it from C:\Program Files\dotnet\host\fxr\6.0.0-preview.3.21201.4\hostfxr.dll failed

1
Answered Apr 14 '21 at 03:59
avatar  of snickler
snickler

Another nice suggestion would be to allow an option to include both arm64/x64 hostfxr.dlls in with framework-dependent deployments. Procmon lead me to discover that upon the load of a .NET executable, it first checks the directory the application exists for a hostfxr.dll.

By copying an x64 hostfxr.dll to the same directory of an x64 targeted framework-dependent app and running the executable, it properly loaded (In reference to https://github.com/microsoft/PowerToys/issues/10658)

This is more of a short-term workaround than an actual solution, though.

1
Answered Apr 14 '21 at 04:11
avatar  of snickler
snickler

As you suggest, this option is more of a short term workaround. We will get this working so that this suggestion isn't needed.

Also, it wouldn't work for cross platform apps on one hand (they would need a lot of hostfxr binaries) and it wouldn't match with apps that were R2R compiled.

1
Answered Apr 14 '21 at 04:30
avatar  of richlander
richlander

Can someone catch me up on why this is needed? Why would I want x64 dotnet on my arm64 computer, once a native version is available?

I'm sure this completely outs me as a noob, but why would installing to the same path be a bad idea? So that the two architectures replace each other?

1
Answered Apr 14 '21 at 13:49
avatar  of Marv51
Marv51

Can someone catch me up on why this is needed? Why would I want x64 dotnet on my arm64 computer, once a native version is available?

I'm sure this completely outs me as a noob, but why would installing to the same path be a bad idea? So that the two architectures replace each other?

x64 on Arm64 emulation exists currently for the Windows Insiders Dev Channel builds, so we're able to run both Arm64 and x64 applications side by side. The current structure of the .NET installation throws a wrench in the mix for being able to execute .NET Core/.NET 5+ applications that are either of those architectures - since they both install to C:\Program Files\dotnet by default.

Since the same directory is being used, you're likely to have multiple SDK versions and packs installed with no way of telling whether they're x64 or Arm64. When dotnet loads the native hostfxr.dll (.NET Host Resolver) of one of the runtimes installed and it doesn't match the architecture of the calling executable (either dotnet.exe or the compiled-app.exe), you'll immediately encounter the error I listed above.

Basically, everything being loaded by either the dotnet executable or the executable/dll produced after building your project, and the .NET runtime dlls need to be built in the same architecture.

It's the same as if you had an x64 executable on an x64 build of Windows trying to load an ARM64 dll.

1
Answered Apr 14 '21 at 14:27
avatar  of snickler
snickler