Unable to migrate data from AWS S3 bucket to Azure Blob storage

[shaikjahangir04]

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

10.12.2

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

.\azcopy copy "https://aispl-testing.s3.ap-south-1.amazonaws.com/testing/" "https://datamigrationstracct.blob.core.windows.net/data-migration-container?" --recursive=true --s2s-handle-invalid-metadata=RenameIfInvalid

What problem was encountered?

Skipping S3 object testing/._.TESTT$1112122_____8u.tyYYY20.txt, as it is not a valid Blob name. Rename the object and retry the transfer

How can we reproduce the problem in the simplest way?

Have you found a mitigation/solution?

[shaikjahangir04]

@siminsavani-msft do we have any update on this? kindly assist me on the reported issue

[siminsavani-msft]

Hi @shaikjahangir04 ! When I use AzCopy to upload files locally to Azure with the same name, it is successful. Could you please attach the logs for this issue?

[shaikjahangir04]

Azcopy logs.zip
Kindly find the attached logs, I'm also able to copy it from local but unable to make it from S3

[mohsha-msft]

Hey @shaikjahangir04 ,

I was looking at the logs and I found following :-

COPYFAILED: https://aispl-testing.s3.ap-south-1.amazonaws.com/testing : 000 : The specified key does not exist. Dst: https://datamigrationstracct.blob.core.windows.net/data-migration-container?SAS

COPYSUCCESSFUL: https://datamigrationstracct.blob.core.windows.net/data-migration-container/testing/_TESTT$11--12122_____8u.tyYYY20.txt

It seems the transfer was indeed successful. Can you please check the container once to verify the files?

[mohsha-msft]

As for the other log,

Skipping S3 object testing/._.TESTT.txt, as it is not a valid Blob name. Rename the object and retry the transfer
The given name is not a valid blob name so the transfer skipped this.

[shaikjahangir04]

I can transfer the same file if I initiate it from my local desktop path, it is failing only if i do it from AWS S3. also I have added the switch --s2s-handle-invalid-metadata=RenameIfInvalid so it should rename the file.

[mohsha-msft]

I see. So, can you send me log of your local->Azure transfer of the same file (._.TESTT.txt) ? Also, can you verify once that the file is present in the destination?

[shaikjahangir04]

AZCOPY Local copy logs.log
I cannot see the file on the destination, attached the screenshot and local copy logs

[shaikjahangir04]

I tried initiating the copy from local drive and I can make it successful, I cannot copy the same file from AWS S3. Also I have lakhs of files with the file name starting with ._. in aws and it is very difficult to rename all the files.

[mohsha-msft]

Thanks for your inputs.
Yes, I can see it transferred successfully from Local -> Azure. Let me debug why it didn't success directly from AWS -> Azure.

[mohsha-msft]

azure-storage-azcopy/cmd/zc_traverser_s3.go

Lines 66 to 75 in 058bd5b

	func (t *s3Traverser) Traverse(preprocessor objectMorpher, processor objectProcessor, filters []ObjectFilter) (err error) {
	invalidAzureBlobName := func (objectKey string) bool {
	/* S3 object name is invalid if it ends with period or
	one of virtual directories in path ends with period.
	This list is not exhaustive
	*/
	return strings.HasSuffix(objectKey, ".") ||
	strings.Contains(objectKey, "/.")
	}
	invalidNameErrorMsg := "Skipping S3 object %s, as it is not a valid Blob name. Rename the object and retry the transfer"

Issue with the traverser function. @siminsavani-msft let's take this up in next weekly.

[zezha-msft]

@mohsha-msft I think @nakulkar-msft had a reason to not allow the prefixed periods. Remember the service API to upload and copy are different and may behave differently too. My memory is failing me here, but I think if you specified prefixed periods in the source URL of the copy API, they get chopped off by the Storage service. Could you please try a copy request using PostMan?

[mohsha-msft]

Hey @shaikjahangir04,

I've added a fix for your problem. Can you please try this drop and let me know your feedback?
If this solves your problem, I'll publish these changes in next release of AzCopy. (10.14.0)

[shaikjahangir04]

7da9ac97-44b2-ae40-689a-4053d9c0b3af.log
Hi Mohit,

I can transfer the files with ._.* from S3 to azure blob using the package which you have provided. attached the logs for your reference.

[mohsha-msft]

Hey @shaikjahangir04 ,

I'm really happy to hear that. Please keep trying and verify the changes on the destination as well. If you face any issue, reach out to us on this channel itself.

[mohsha-msft]

Fix released in AzCopy 10.14.0