How to move existing solution to scaffold setup

[halcwb]

It would be nice to have a guide how to move an existing solution to the scaffold structure. At the moment I am trying to do this by trial and error.

[Vidarls]

Not sure there is a single correct way of doing that..

My procedure is quite simple:

1. run initial scaffold build to initiate solution and stuff
2. Copy projects from existing solution to src folder
3. Copy test projects from existing solution to test folder
4. Add projects to newly created scaffold solution using add existing project
5. Run .\paket\paket.exe convert--from--nuget hard (think it is hard, always need to check if it is hard or force when paket.dependencies are already present). To convert nuget projects to paket.
6. Run build to see that all is well
7. Fix broken stuff
8. Push to central repo
9. Remember that you now have a release notes file that controls versioning
10. Update release notes file
11. Push once more

Done, but this only covers my use, not sure it is generic enough to use as a recipe for others.

[halcwb]

That would be a way, only, I already have a solution on github with a src folder, i.e. the structure matches the scaffold setup.

[Vidarls]

I've done the same procedure with existing projects with almost matching structure, and since git operates on file content, it usually creates nice diffs, even if your solution and folders are not the same.

But again, my experience may not apply to others.

[halcwb]

I am still having trouble getting it working with a multi project solution. Seems that the current setup is geared toward a single project solution. I have modified the build script to suite my needs but the build target now fails. The current script I am using looks like:

// --------------------------------------------------------------------------------------
// FAKE build script
// --------------------------------------------------------------------------------------

#r @"packages/FAKE/tools/FakeLib.dll"
open Fake
open Fake.Git
open Fake.AssemblyInfoFile
open Fake.ReleaseNotesHelper
open System
open System.IO
#if MONO
#else
#load "packages/SourceLink.Fake/tools/Fake.fsx"
open SourceLink
#endif

// --------------------------------------------------------------------------------------
// START TODO: Provide project-specific details below
// --------------------------------------------------------------------------------------

// Information about the project are used
//  - for version and project name in generated AssemblyInfo file
//  - by the generated NuGet package
//  - to run tests and to publish documentation on GitHub gh-pages
//  - for documentation, you also need to edit info in "docs/tools/generate.fsx"

// The name of the project
// (used by attributes in AssemblyInfo, name of a NuGet package and directory in 'src')
let projects = [
    ("Informedica.Settings.Services", "Get, Add, and Delete Settings")
    ("Informedica.GenImport.Services", "Import G-Standard")
    ("Informedica.GenPDMS.Services", "Interface with PDMS")
    ("Informedica.GenForm.Services", "Deliver Decision Support")
    ("Informedica.GenPres.Services", "Create Prescriptions")
    ("Informedica.GenPres.MvcWeb", "GenPres UserInterface")
]


// List of author names (for NuGet package)
let authors = [ "halcwb" ]

// Tags for your project (for NuGet package)
let tags = ""

// File system information 
let solutionFile  = "GenPres.sln"

// Pattern specifying assemblies to be tested using NUnit
let testAssemblies = "tests/**/bin/Release/*Tests*.dll"

// Git configuration (used for publishing documentation in gh-pages branch)
// The profile where the project is posted
let gitOwner = "halcwb" 
let gitHome = "https://github.com/" + gitOwner

// The name of the project on GitHub
let gitName = "GenPres"

// The url for the raw files hosted
let gitRaw = environVarOrDefault "gitRaw" "https://raw.github.com/halcwb"

// --------------------------------------------------------------------------------------
// END TODO: The rest of the file includes standard build steps
// --------------------------------------------------------------------------------------

// Read additional information from the release notes document
let release = LoadReleaseNotes "RELEASE_NOTES.md"

// Helper active pattern for project types
let (|Fsproj|Csproj|Vbproj|) (projFileName:string) = 
    match projFileName with
    | f when f.EndsWith("fsproj") -> Fsproj
    | f when f.EndsWith("csproj") -> Csproj
    | f when f.EndsWith("vbproj") -> Vbproj
    | _                           -> failwith (sprintf "Project file %s not supported. Unknown project type." projFileName)

// Generate assembly info files with the right version & up-to-date information
Target "AssemblyInfo" (fun _ ->

    for project, summary in projects do

        let getAssemblyInfoAttributes projectName =
            [ Attribute.Title (projectName)
              Attribute.Product project
              Attribute.Description summary
              Attribute.Version release.AssemblyVersion
              Attribute.FileVersion release.AssemblyVersion ]

        let getProjectDetails projectPath =
            let projectName = System.IO.Path.GetFileNameWithoutExtension(projectPath)
            ( projectPath, 
              projectName,
              System.IO.Path.GetDirectoryName(projectPath),
              (getAssemblyInfoAttributes projectName)
            )

        !! "src/**/*.??proj"
        |> Seq.map getProjectDetails
        |> Seq.iter (fun (projFileName, projectName, folderName, attributes) ->
            try
                match projFileName with
                | Fsproj -> CreateFSharpAssemblyInfo (folderName @@ "AssemblyInfo.fs") attributes
                | Csproj -> CreateCSharpAssemblyInfo ((folderName @@ "Properties") @@ "AssemblyInfo.cs") attributes
                | Vbproj -> CreateVisualBasicAssemblyInfo ((folderName @@ "My Project") @@ "AssemblyInfo.vb") attributes
            with 
            | _ -> ()
        )
    )

// Copies binaries from default VS location to exepcted bin folder
// But keeps a subdirectory structure for each project in the 
// src folder to support multiple project outputs
Target "CopyBinaries" (fun _ ->
    !! "src/**/*.??proj"
    |>  Seq.map (fun f -> ((System.IO.Path.GetDirectoryName f) @@ "bin/Release", "bin" @@ (System.IO.Path.GetFileNameWithoutExtension f)))
    |>  Seq.iter (fun (fromDir, toDir) -> CopyDir toDir fromDir (fun _ -> true))
)

// --------------------------------------------------------------------------------------
// Clean build results

Target "Clean" (fun _ ->
    CleanDirs ["bin"; "temp"]
)

Target "CleanDocs" (fun _ ->
    CleanDirs ["docs/output"]
)

// --------------------------------------------------------------------------------------
// Build library & test project

Target "Build" (fun _ ->
    for project, _ in projects do
        !! ("src/**/" + project)
        |> MSBuildRelease null "Build"
        |> Log (project + "\n: ")
)

// --------------------------------------------------------------------------------------
// Run the unit tests using test runner

Target "RunTests" (fun _ ->
    !! testAssemblies
    |> NUnit (fun p ->
        { p with
            DisableShadowCopy = true
            TimeOut = TimeSpan.FromMinutes 20.
            OutputFile = "TestResults.xml" }
    )
)

#if MONO
#else
// --------------------------------------------------------------------------------------
// SourceLink allows Source Indexing on the PDB generated by the compiler, this allows
// the ability to step through the source code of external libraries https://github.com/ctaggart/SourceLink

Target "SourceLink" (fun _ ->

    for project, _ in projects do

        let baseUrl = sprintf "%s/%s/{0}/%%var2%%" gitRaw (project.ToLower())
        use repo = new GitRepo(__SOURCE_DIRECTORY__)

        let addAssemblyInfo (projFileName:String) = 
            match projFileName with
            | Fsproj -> (projFileName, "**/AssemblyInfo.fs")
            | Csproj -> (projFileName, "**/AssemblyInfo.cs")
            | Vbproj -> (projFileName, "**/AssemblyInfo.vb")

        !! "src/**/*.??proj"
        |> Seq.map addAssemblyInfo
        |> Seq.iter (fun (projFile, assemblyInfo) ->
            let proj = VsProj.LoadRelease projFile 
            logfn "source linking %s" proj.OutputFilePdb
            let files = proj.Compiles -- assemblyInfo
            repo.VerifyChecksums files
            proj.VerifyPdbChecksums files
            proj.CreateSrcSrv baseUrl repo.Revision (repo.Paths files)
            Pdbstr.exec proj.OutputFilePdb proj.OutputFilePdbSrcSrv
        )
)

#endif

// --------------------------------------------------------------------------------------
// Build a NuGet package

Target "NuGet" (fun _ ->
    Paket.Pack(fun p -> 
        { p with
            OutputPath = "bin"
            Version = release.NugetVersion
            ReleaseNotes = toLines release.Notes})
)

//Target "PublishNuget" (fun _ ->
//    Paket.Push(fun p -> 
//        { p with
//            WorkingDir = "bin" })
//)


// --------------------------------------------------------------------------------------
// Generate the documentation

Target "GenerateReferenceDocs" (fun _ ->
    if not <| executeFSIWithArgs "docs/tools" "generate.fsx" ["--define:RELEASE"; "--define:REFERENCE"] [] then
      failwith "generating reference documentation failed"
)

let generateHelp' fail debug =
    let args =
        if debug then ["--define:HELP"]
        else ["--define:RELEASE"; "--define:HELP"]
    if executeFSIWithArgs "docs/tools" "generate.fsx" args [] then
        traceImportant "Help generated"
    else
        if fail then
            failwith "generating help documentation failed"
        else
            traceImportant "generating help documentation failed"

let generateHelp fail =
    generateHelp' fail false

Target "GenerateHelp" (fun _ ->
    DeleteFile "docs/content/release-notes.md"
    CopyFile "docs/content/" "RELEASE_NOTES.md"
    Rename "docs/content/release-notes.md" "docs/content/RELEASE_NOTES.md"

    DeleteFile "docs/content/license.md"
    CopyFile "docs/content/" "LICENSE.txt"
    Rename "docs/content/license.md" "docs/content/LICENSE.txt"

    generateHelp true
)

Target "GenerateHelpDebug" (fun _ ->
    DeleteFile "docs/content/release-notes.md"
    CopyFile "docs/content/" "RELEASE_NOTES.md"
    Rename "docs/content/release-notes.md" "docs/content/RELEASE_NOTES.md"

    DeleteFile "docs/content/license.md"
    CopyFile "docs/content/" "LICENSE.txt"
    Rename "docs/content/license.md" "docs/content/LICENSE.txt"

    generateHelp' true true
)

Target "KeepRunning" (fun _ ->    
    use watcher = new FileSystemWatcher(DirectoryInfo("docs/content").FullName,"*.*")
    watcher.EnableRaisingEvents <- true
    watcher.Changed.Add(fun e -> generateHelp false)
    watcher.Created.Add(fun e -> generateHelp false)
    watcher.Renamed.Add(fun e -> generateHelp false)
    watcher.Deleted.Add(fun e -> generateHelp false)

    traceImportant "Waiting for help edits. Press any key to stop."

    System.Console.ReadKey() |> ignore

    watcher.EnableRaisingEvents <- false
    watcher.Dispose()
)

Target "GenerateDocs" DoNothing

let createIndexFsx lang =
    let content = """(*** hide ***)
// This block of code is omitted in the generated HTML documentation. Use 
// it to define helpers that you do not want to show in the documentation.
#I "../../../bin"

(**
F# Project Scaffold ({0})
=========================
*)
"""
    let targetDir = "docs/content" @@ lang
    let targetFile = targetDir @@ "index.fsx"
    ensureDirectory targetDir
    System.IO.File.WriteAllText(targetFile, System.String.Format(content, lang))


Target "AddLangDocs" (fun _ ->
    let args = System.Environment.GetCommandLineArgs()
    if args.Length < 4 then
        failwith "Language not specified."

    args.[3..]
    |> Seq.iter (fun lang ->
        if lang.Length <> 2 && lang.Length <> 3 then
            failwithf "Language must be 2 or 3 characters (ex. 'de', 'fr', 'ja', 'gsw', etc.): %s" lang

        let templateFileName = "template.cshtml"
        let templateDir = "docs/tools/templates"
        let langTemplateDir = templateDir @@ lang
        let langTemplateFileName = langTemplateDir @@ templateFileName

        if System.IO.File.Exists(langTemplateFileName) then
            failwithf "Documents for specified language '%s' have already been added." lang

        ensureDirectory langTemplateDir
        Copy langTemplateDir [ templateDir @@ templateFileName ]

        createIndexFsx lang)
)

// --------------------------------------------------------------------------------------
// Release Scripts

Target "ReleaseDocs" (fun _ ->
    let tempDocsDir = "temp/gh-pages"
    CleanDir tempDocsDir
    Repository.cloneSingleBranch "" (gitHome + "/" + gitName + ".git") "gh-pages" tempDocsDir

    CopyRecursive "docs/output" tempDocsDir true |> tracefn "%A"
    StageAll tempDocsDir
    Git.Commit.Commit tempDocsDir (sprintf "Update generated documentation for version %s" release.NugetVersion)
    Branches.push tempDocsDir
)

#load "paket-files/fsharp/FAKE/modules/Octokit/Octokit.fsx"
open Octokit

Target "Release" (fun _ ->
    StageAll ""
    Git.Commit.Commit "" (sprintf "Bump version to %s" release.NugetVersion)
    Branches.push ""

    Branches.tag "" release.NugetVersion
    Branches.pushTag "" "origin" release.NugetVersion

    // release on github
    createClient (getBuildParamOrDefault "github-user" "") (getBuildParamOrDefault "github-pw" "")
    |> createDraft gitOwner gitName release.NugetVersion (release.SemVer.PreRelease <> None) release.Notes 
    // TODO: |> uploadFile "PATH_TO_FILE"    
    |> releaseDraft
    |> Async.RunSynchronously
)

Target "BuildPackage" DoNothing

// --------------------------------------------------------------------------------------
// Run all targets by default. Invoke 'build <Target>' to override

Target "All" DoNothing

"Clean"
  ==> "AssemblyInfo"
  ==> "Build"
  ==> "CopyBinaries"
  ==> "RunTests"
  =?> ("GenerateReferenceDocs",isLocalBuild)
  =?> ("GenerateDocs",isLocalBuild)
  ==> "All"
  =?> ("ReleaseDocs",isLocalBuild)

"All" 
#if MONO
#else
  =?> ("SourceLink", Pdbstr.tryFind().IsSome )
#endif
  ==> "NuGet"
  ==> "BuildPackage"

"CleanDocs"
  ==> "GenerateHelp"
  ==> "GenerateReferenceDocs"
  ==> "GenerateDocs"

"CleanDocs"
  ==> "GenerateHelpDebug"

"GenerateHelp"
  ==> "KeepRunning"

"ReleaseDocs"
  ==> "Release"

//"BuildPackage"
//  ==> "PublishNuget"
//  ==> "Release"

RunTargetOrDefault "All"

However, this runs in a:

1. System.Exception: Invalid tag structure. Trying to close target tag but sta
   ck is [task; target]
   at Fake.TraceHelper.closeTag@123.Invoke(String message) in C:\code\fake\src\a
   pp\FakeLib\TraceHelper.fs:line 123
   at Fake.TraceHelper.closeTag(String tag) in C:\code\fake\src\app\FakeLib\Trac
   eHelper.fs:line 123
   at Fake.TraceHelper.traceEndTarget(String name) in C:\code\fake\src\app\FakeL
   ib\TraceHelper.fs:line 139
   at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target) in C:\code\fake
   \src\app\FakeLib\TargetHelper.fs:line 413

[Vidarls]

Any reason for not just building your .sln file rather than individual proj files?

[halcwb]

The reason is that the projects in the solution are services that work with each other, so, I would like to have more fine grained control. However, I can build the solution as a whole, however, I then still run into errors. When I build the solution manually (doing a rebuild), it works. Trying to build the solution by the build script runs into errors. Unfortunately, there is no real logging, and the command window overflows. At the moment I seem to have a problem with paket:

1. Building c:\Users\hal\Dropbox\Development\GenPres\GenPres.sln failed with e
   xitcode 1.
2. MSB3073: c:\Users\hal\Dropbox\Development\GenPres.paket\paket.targets(36,5
   ): The command ""c:\Users\hal\Dropbox\Development\GenPres.paket\paket.exe" rest
   ore --references-files "c:\Users\hal\Dropbox\Development\GenPres\src\Informedica
   .GenPres.Services\paket.references"" exited with code 1.
3. MSB3073: c:\Users\hal\Dropbox\Development\GenPres.paket\paket.targets(36,5
   ): The command ""c:\Users\hal\Dropbox\Development\GenPres.paket\paket.exe" rest
   ore --references-files "c:\Users\hal\Dropbox\Development\GenPres\tests\Informedi
   ca.Utilities.Tests\paket.references"" exited with code 1.

But the message is not very helpfull ;-(

[Vidarls]

Did you convert from nuget to Paket dependency managmenet?

[halcwb]

Yep.

[halcwb]

This is what I get checking the offending paket command:

1. Building c:\Users\hal\Dropbox\Development\GenPres\GenPres.sln failed with e
   xitcode 1.
2. MSB3073: c:\Users\hal\Dropbox\Development\GenPres.paket\paket.targets(36,5
   ): The command ""c:\Users\hal\Dropbox\Development\GenPres.paket\paket.exe" rest
   ore --references-files "c:\Users\hal\Dropbox\Development\GenPres\tests\Informedi

ca.Utilities.Tests\paket.references"" exited with code 1.

Press any key to continue . . .
c:\Users\hal\Dropbox\Development\GenPres>.paket\paket restore --references-files
.\tests\Informedica.Utilities.Tests\paket.references
Paket version 0.39.2.0
0 seconds - ready.

c:\Users\hal\Dropbox\Development\GenPres>

So, when paket runs the command from the build script, it fails, but when doing this manually it just runs fine. Could this be some sort of race condition, in which access is denied to the paket.references file?

[Vidarls]

Now I'm blank. @forki any idea?

[forki]

fsprojects/Paket#762?

[halcwb]

Thanks to @forki, I was able to run the build with:

Target "Build" (fun _ ->
    !! "GenPres.sln"
    |> MSBuildReleaseExt "" [("RestorePackages", "False")] "Build"
    |> ignore
)

So, disabling the restore process during build seems to fix the issue. However, moving a multiproject sotion to the scaffold still has some challenges, I'll come back to that.

[forki]

("RestorePackages", "False") should be set automatically in FAKE 3.38 - can you please try to verify that?

[forki]

@Thorium: does `FAKE 3.38 work for your scenario?

[Thorium]

I actually still use plain MSBuild Xml and not Fake... So I did set it just RestorePackages false as a property and get (only) a minute away from build.

[dsyme]

Closing this discussion - the original scenario of "moving" is not really supported by the scaffold