[WIP] Add HDInsight resource support #924
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jjcollinge
changed the title
|
@jjcollinge if you plan to support only linux cluster you could simply the structure due to the end of support of window cluster in few month |
tombuildsstuff
suggested changes
Mar 15, 2018
There was a problem hiding this comment.
hey @jjcollinge
Thanks for this PR - apologies for the delay in reviewing this! I've taken a look through and left some comments in-line, but this is off to a good start :)
Given this is a large resource, I'd suggest it might be worth focusing only on the Required fields/blocks first and then add any Optional fields in a follow up PR?
Thanks!
azurerm/resource_arm_hdinsight.go
Outdated
| }, false), | ||
| }, | ||
|
|
||
| //TODO: Add support for Windows |
There was a problem hiding this comment.
given Windows support is EOL'd (and disappearing in a few months) - I think it'd be sensible to just focus on Linux at this point?
azurerm/resource_arm_hdinsight.go
Outdated
| "cluster_version": { | ||
| Type: schema.TypeString, | ||
| Optional: true, | ||
| Default: "~3.6", |
There was a problem hiding this comment.
given that (presumably) this changes as a new version is released, I think it'd be sensible to remove this default value and make this required
azurerm/resource_arm_hdinsight.go
Outdated
| "Linux", | ||
| }, false), | ||
| DiffSuppressFunc: ignoreCaseDiffSuppressFunc, | ||
| }, |
There was a problem hiding this comment.
(in which case, can we remove this field)
azurerm/resource_arm_hdinsight.go
Outdated
| Type: schema.TypeString, | ||
| Optional: true, | ||
| }, | ||
| "rest_auth_credential__password": { |
There was a problem hiding this comment.
can we fix the double underscores in these three fields -> a single underscore?
azurerm/resource_arm_hdinsight.go
Outdated
| }, | ||
| "min_instance_count": { | ||
| Type: schema.TypeInt, | ||
| Optional: true, |
There was a problem hiding this comment.
does this want some validation and a default value?
| } | ||
|
|
||
| resource "azurerm_storage_account" "test" { | ||
| name = "hdi%d" |
There was a problem hiding this comment.
there's a limit on the number of chars available for a storage account name; could we make this a random string of 4-6 chars + with a acctestsa prefix
| } | ||
|
|
||
| resource "azurerm_hdinsight_cluster" "test" { | ||
| name = "hdi%d" |
There was a problem hiding this comment.
could we make this acctesthdi%d
| "checksumSHA1": "OXG8AijhiNimr1zNSJ85SnfcrI8=", | ||
| "path": "github.com/Azure/azure-sdk-for-go/services/hdinsight/mgmt/2015-03-01-preview/hdinsight", | ||
| "revision": "7b986a3a0511159a0a7877f0d7d775bdfe916d29", | ||
| "revisionTime": "2017-12-16T00:10:16Z" |
There was a problem hiding this comment.
can we pin to using v12.5.0-beta of the Azure SDK for the moment? we can do that via:
govendor update github.com/Azure/azure-sdk-for-go/services/hdinsight/mgmt/2015-03-01-preview/hdinsight.@=v12.5.0-beta
| "checksumSHA1": "0f2bCAIQjO9op810x5LnuZnAmM0=", | ||
| "path": "github.com/Azure/azure-sdk-for-go/version", | ||
| "revision": "1e334c402ea1460704b0263e5d188f28ad946ce1", | ||
| "revisionTime": "2018-02-16T17:41:56Z" |
There was a problem hiding this comment.
(which should mean this isn't needed); although we plan to upgrade in the near future #825
azurerm/config.go
Outdated
| setUserAgent(&hdInsightClustersClient.Client) | ||
| hdInsightClustersClient.Authorizer = auth | ||
| hdInsightClustersClient.Sender = sender | ||
| hdInsightClustersClient.SkipResourceProviderRegistration = c.skipProviderRegistration |
There was a problem hiding this comment.
minor could we switch over to using the new registration format? c.configureClient(&hdInsightClustersClient.Client, auth) (we're doing this gradually for all new resources)
|
Acceptance tests still failing with DIFF errors: Current |
|
@tombuildsstuff - any ideas how I can get around the DIFF issue posted above? |
|
@jjcollinge apologies - I'd missed I'd not replied to this, I've put some time aside tomorrow to take a look. |
This comment has been minimized.
This comment has been minimized.
|
hey @jjcollinge Apologies for the delayed response here - I've spent some time looking into HDInsights over the past few days and have taken the learnings from this PR but taken a slightly different approach which has culminated in PR #1267. I hope you don't mind, but I'm going to close this one in favour of that one. Thanks! |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks! |
ghost
locked and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for creating Azure HDInsight clusters. The PR is not finished but it is in a working state i.e. clusters can be created, read, updated and deleted. I wanted to create the PR early to get some feedback and to help make it idiomatic whilst I finish it off. Currently, the acceptance tests are failing for 2 reasons:
Computed: truebut then got an error when trying to set them - as they're effectively required fields.Some help on addressing the above issues would be great!
I also do some 'experimental' mapping between keys i.e.
restCredentialAuth.isEnabled-->rest_credential_auth__is_enabled. This probably needs changing at some point but not sure on the best way to handle dynamic maps. I tried just embedding a JSON string for the configuration but I couldn't get it building.HDInsight can be configured in a number of ways so I do plan to write some additional tests for Spark, HBase, Hive, etc.
Appreciate any thoughts at this stage.