Special characters encoding issue

[bartvermeersch]

Can someone point me whether this issue is caused by the CICD process or by Microsoft365DSC itself?

When applying this DSC, the é character becomes é

@{
    Config_AzureAD = @{
    }
    NonNodeData = @{
        AzureAD = @{
            Users = @(
            @{
                UniqueID            = "Fred.Smith2"
                UserPrincipalName  = "Fred.Smith2@{{TenantId}}"
                FirstName          = "Frédéric"
                LastName           = "Smith"
                DisplayName        = "Frédéric J. Smith"
                Ensure             = "Present"
            }

            )
        }
    }
}

Frédéric is rendered as Frédéric

:32.7580117Z VERBOSE: [fv-az899-25]: LCM:  [ Start  Resource ]  
:32.7590683Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration]
:32.7595264Z VERBOSE: [fv-az899-25]: LCM:  [ Start  Test     ]  
:32.7601767Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration]
:32.8209316Z VERBOSE: [fv-az899-25]:                            
:32.8220614Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] Testing configuration of 
:32.8225314Z Azure AD User Fred.Smith2@mytenant.com
:32.8235164Z VERBOSE: [fv-az899-25]:                            
:32.8239543Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] Getting configuration of 
:32.8246096Z Office 365 User Fred.Smith2@mytenant.com
:33.7070529Z VERBOSE: [fv-az899-25]:                            
:33.7081710Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] Getting Office 365 User 
:33.7085204Z Fred.Smith2@mytenant.com
:33.7500688Z VERBOSE: [fv-az899-25]:                            
:33.7512909Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] The specified User doesn't 
:33.7517496Z already exist.
:33.7528349Z VERBOSE: [fv-az899-25]:                            
:33.7533540Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] Current Values: 
:33.7611257Z DisplayName=$null
:33.7633370Z Ensure=Absent
:33.7657875Z FirstName=$null
:33.7665297Z LastName=$null
:33.7760972Z UserPrincipalName=$null
:33.7763235Z VERBOSE: [fv-az899-25]:                            
:33.7779000Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] Target Values: 
:33.7865617Z DisplayName=Frédéric J. Smith
:33.7889723Z Ensure=Present
:33.7903915Z FirstName=Frédéric
:33.7917125Z LastName=Smith
:33.7976809Z UserPrincipalName=Fred.Smith2@mytenant.com
:33.7992970Z Verbose=True
:34.3136176Z VERBOSE: [fv-az899-25]:                            
:34.3149301Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] Test-TargetResource returned 
:34.3154633Z False
:34.3165195Z VERBOSE: [fv-az899-25]: LCM:  [ End    Test     ]  
:34.3176093Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration]  in 1.4850 seconds.
:34.3186687Z VERBOSE: [fv-az899-25]: LCM:  [ Start  Set      ]  
:34.3197438Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration]
:34.3213652Z VERBOSE: [fv-az899-25]:                            
:34.3219230Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] Setting configuration of 
:34.3225629Z Office 365 User Fred.Smith2@mytenant.com
:34.3912936Z VERBOSE: [fv-az899-25]:                            
:34.3924482Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] Getting configuration of 
:34.3928188Z Office 365 User Fred.Smith2@mytenant.com
:34.5647666Z VERBOSE: [fv-az899-25]:                            
:34.5659024Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] Getting Office 365 User 
:34.5662899Z Fred.Smith2@mytenant.com
:34.6125073Z VERBOSE: [fv-az899-25]:                            
:34.6136531Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] The specified User doesn't 
:34.6141422Z already exist.
:35.0528304Z VERBOSE: [fv-az899-25]:                            
:35.0544581Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] Creating Office 365 User 
:35.0548699Z Fred.Smith2@mytenant.com
:35.0591231Z VERBOSE: [fv-az899-25]:                            
:35.0596808Z [[AADUser]AADUser-Fred.Smith2@mytenant.com::[AzureAD]AzureAD_Configuration] Creating new user with 
:35.0612018Z values: AccountEnabled=True
:35.0659483Z DisplayName=Frédéric J. Smith
:35.0672178Z GivenName=Frédéric
:35.0759288Z UserPrincipalName=Fred.Smith2@xxxxx
:35.5288503Z VERBOSE: [fv-az899-25]: LCM:  [ End    Set      ] 

I checked the psd1 file and it is UTF8withBOM

Preamble          :
BodyName          : utf-8
EncodingName      : Unicode (UTF-8)
HeaderName        : utf-8
WebName           : utf-8
WindowsCodePage   : 1200
IsBrowserDisplay  : True
IsBrowserSave     : True
IsMailNewsDisplay : True
IsMailNewsSave    : True
IsSingleByte      : False
EncoderFallback   : System.Text.EncoderReplacementFallback
DecoderFallback   : System.Text.DecoderReplacementFallback
IsReadOnly        : True
CodePage          : 65001

[ykuijs]

Hi @bartvermeersch, is this change also reflected in the object in Microsoft 365 or just in the logging?

Background: This is caused by different encoding of files. Different applications use different encodings, resulting in characters being interpreted incorrectly. CodeX means something different in a different encoding.

[bartvermeersch]

Also in the object I'm afraid.

[bartvermeersch]

I got a bit further, all files are UTF8BOM, but the generated MOF is UTF-16 LE

[bartvermeersch]

I'm stuck, I added some logging:

Write-Log -Object (Get-Content -Path (Resolve-Path -Path $finalPsdPath).Path -Raw)
At the end of the prebuilt script, it returns the é characters correctly (D:\a\1\s\Output\envxxx\envxxx.psd1)

At the beginning of the built script, the same file returns é characters?

What am I missing?

[bartvermeersch]

I was able to fix the encoding of the MOF by removing

pwsh: true

from the Pre Build task.

https://github.com/ykuijs/M365DSC_CICD/blob/9035794433e9e0eb43228f309055599ffe4ca3e7/Pipelines/build-template.yaml#L66C11-L66C21

Unfortunately, the final outcome remans wrong. In deploy.ps1 the MOF is UT8 and contains é characters (perfect), but the Start-DscConfiguration returns é instead...