ps script works in AWSPowerShell.NetCore 4.1.632 but not in newer version

[mmckenzie25]

Hi All,
I have a powershell script that copies 3 files to 3 different folders on an s3.
We have been using 4.1.632 and the script runs fine.
We upgraded the modules to latest (not preview) and the script fails. When I strip the code down and send 1 file directly it works. The minute I add the for loop back in it fails.
The error message is the same for each copy:
ERROR: Write-S3Object : Access Denied
ERROR: At line:41 char:9
ERROR: + Write-S3Object -BucketName $bucketName -Key $destinationKey - ...
ERROR: + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ERROR: + CategoryInfo : NotSpecified: (:) [Write-S3Object], AmazonS3Exception
ERROR: + FullyQualifiedErrorId : Amazon.S3.AmazonS3Exception,Amazon.PowerShell.Cmdlets.S3.WriteS3ObjectCmdlet
ERROR:

Here is a redacted version of the script that runs in 4.1.632 and not in the current version:

Set-AWSCredential -AccessKey -SecretKey
$targetFolder = "\Server\share\folder1\folder2\f3\f4\F5\Folder6\ToMAS8"
$bucketName = "masms-22-cos-files-name-pr-dal-cust-files"

Set parameters

$endpointUrl = "https://s3.us-south.cloud-object-storage.appdomain.cloud"
$log = "E:\Logs\ToMAS8.log"

Import the AWS SDK for .NET (assuming the AWS PowerShell module is already installed)

#Import-Module AWS.Tools.S3 #-tried using the newer ASW.Tools but it has the same error
Import-Module AWSPowerShell.NetCore

Function to log messages

function Log-Message {
param (
[string]$message
)
$timestamp = (Get-Date).ToString("yyyy-MM-dd HH:mm:ss")
Add-Content -Path $log -Value "$timestamp - $message"
}

Define the folders and their associated names in S3

$folderNames = @("CRAFT", "LABOR", "PERSON")

Get all XML files from the root directory

$xmlFiles = Get-ChildItem -Path $targetFolder -Filter "*.xml" | Select-Object -First 3
foreach ($file in $xmlFiles) {
# Determine the appropriate S3 folder (CRAFT, LABOR, PERSON) based on the file name
$folderName = $null
if ($file.Name -like "CRAFT") {
$folderName = "CRAFT"
} elseif ($file.Name -like "LABOR") {
$folderName = "LABOR"
} elseif ($file.Name -like "PERSON") {
$folderName = "PERSON"
}

if ($folderName) {
    $destinationKey = "/INT538_Maximo_HCM_Data/$folderName/$($file.Name)"
    Write-S3Object -BucketName $bucketName -Key $destinationKey -File $file.FullName -EndpointUrl $endpointUrl
    Log-Message "Copied $($file.FullName) to s3://$bucketName/$destinationKey"
} else {
    Log-Message "Skipped $($file.FullName) - No matching folder for file."
}

}

Get .temp files from the root directory

$tempFiles = Get-ChildItem -Path $targetFolder -Filter "*.temp" | Select-Object -First 3
foreach ($file in $tempFiles) {
# Determine the appropriate S3 folder for .temp files
$folderName = $null
if ($file.Name -like "CRAFT") {
$folderName = "CRAFT"
} elseif ($file.Name -like "LABOR") {
$folderName = "LABOR"
} elseif ($file.Name -like "PERSON") {
$folderName = "PERSON"
}

if ($folderName) {
    $destinationKey = "/FolderName/$folderName/$($file.Name)"
    Write-S3Object -BucketName $bucketName -Key $destinationKey -File $file.FullName -EndpointUrl $endpointUrl
    Log-Message "Copied $($file.FullName) to s3://$bucketName/$destinationKey"
} else {
    Log-Message "Skipped $($file.FullName) - No matching folder for file."
}

}

Delete .temp files from the S3 bucket

$tempObjects = Get-S3Object -BucketName $bucketName -EndpointUrl $endpointUrl | Where-Object { $_.Key -like "*.temp" }

foreach ($object in $tempObjects) {
Remove-S3Object -BucketName $bucketName -Key $object.Key -EndpointUrl $endpointUrl -Force
Log-Message "Deleted s3://$bucketName/$($object.Key)"
}

Log-Message "File transfer and cleanup completed."

Any help is greatly appreciated.

[dscpinheiro]

Versions 4.1.737 and later of the AWS Tools for PowerShell automatically calculate request checksums for a few S3 operations (announcement in #370). This works transparently for the actual S3 service, but there are some 3rd-party implementations that do not support this feature yet (for example #371 - which mentions a similar error message).

Ideally, these 3rd parties should be updated to support the new default integrity protections, but for the moment you can also set the AWS_REQUEST_CHECKSUM_CALCULATION option to WHEN_REQUIRED (this documentation page has more details on the option and what it means: https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html).

[mmckenzie25]

Thank you so much!

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.