azure hosted mac pipeline , suggestion on how to make it faster , currently very slow , on desktop 5 min , on server near 20

Question

i have this pipeline which works great but is very time consuming each build when in desktop xcode it takes less then 5 min to build, in the mac cloud server its takes 20
especlly the parts where it downloads configures and compiles native gems in the
"installing gems and cocoapods script" task bellow
and then there is the Fastlane task bellow : "run fastlane"
where the compilation part is very slow ...
I'm looking for tips and tricks or organize better my pipeline for faster build

    pool:
      vmImage: 'macOS 10.14'
        
    variables:
      scheme: ''
      sdk: 'iphoneos'
      configuration: 'Release'
      
    jobs:
    - job: self_hosted_connect
      timeoutInMinutes: 10
      pool: Default
    
      steps:
        
      - task: CopyFiles@2
        inputs:
          SourceFolder: '$(Agent.HomeDirectory)/../${{parameters.Folderpath}}'
          Contents: '**'
          TargetFolder: '$(build.artifactstagingdirectory)'
      - task: PublishBuildArtifacts@1
        inputs:
          pathToPublish: '$(build.artifactstagingdirectory)'
          artifactName: 'ios_artifacts'
    
    - job: mac_agent
      dependsOn: self_hosted_connect   
      timeoutInMinutes: 30
    
      pool:
        vmImage: 'macOS 10.14'
      steps:
      - script: echo 'Setting up macOS 10.14'
     
      - task: UseRubyVersion@0
        inputs:
         versionSpec: '>= 2.4'
         addToPath: true 
     
      - task: DownloadBuildArtifacts@0
        inputs:
          buildType: 'current'
          downloadType: 'single'
          artifactName: 'ios_artifacts'
          downloadPath: '$(System.ArtifactsDirectory)'  
      
      - script:  |
         
         gem install --no-document bundler
         bundle update --bundler
         bundle install --retry=3 --jobs=4 
         gem install --no-document fastlane
         pod deintegrate
         gem install cocoapods
         pod install    
         pod --version
         
        workingDirectory: '$(System.ArtifactsDirectory)/ios_artifacts'
        displayName: 'installing gems and cocoapods'
      
      - script:  |
         echo 'Start invoking Fastfile'
         fastlane release --verbose
         echo 'Done invoking Fastfile'
        failOnStderr: false 
        workingDirectory: '$(System.ArtifactsDirectory)/ios_artifacts'
        displayName: 'run fastlane'
      
      - task: PublishBuildArtifacts@1
        inputs:
          PathtoPublish: '$(Build.ArtifactStagingDirectory)'
          ArtifactName: 'Artifacts'
          publishLocation: 'Container'
          
    - job: copy_back_files_to_self_hosted_connect
      dependsOn: mac_agent 
      timeoutInMinutes: 30
      pool: Default
      steps:
        - task: DownloadBuildArtifacts@0
          inputs:
            buildType: 'current'
            downloadType: 'single'
            artifactName: 'Artifacts'
            itemPattern: | 
                    **/*.ipa
                    **/*manifest.plist*
            downloadPath: '$(System.ArtifactsDirectory)'
        - task: CopyFiles@2
          inputs:
            SourceFolder: '$(System.ArtifactsDirectory)'
            Contents: | 
                    **/*.ipa
                    **/*manifest.plist*
            TargetFolder: '$(Agent.HomeDirectory)/../${{parameters.FolderCompile}}'

UPDATE USING CACHE

i added the cache section but it gives me an error when i run it

 Resolving key:
2020-10-01T11:39:48.3151700Z  - gems       [string]
2020-10-01T11:39:48.3241700Z  - "Darwin"   [string]
2020-10-01T11:39:48.3720810Z  - my.gemspec [file] (not found)
2020-10-01T11:39:48.3882910Z ##[error]One or more errors occurred. (File not found: my.gemspec)
2020-10-01T11:39:48.4094600Z ##[section]Finishing: Cache gems




    

pool: vmImage: 'macOS 10.14'

variables:
  scheme: ''
  sdk: 'iphoneos'
  configuration: 'Release'
  BUNDLE_PATH: $(Pipeline.Workspace)/.bundle
  
jobs:
- job: self_hosted_connect
  timeoutInMinutes: 10
  pool: Default

  steps:
    
  - task: CopyFiles@2
    inputs:
      SourceFolder: '$(Agent.HomeDirectory)/../${{parameters.Folderpath}}'
      Contents: '**'
      TargetFolder: '$(build.artifactstagingdirectory)'
  - task: PublishBuildArtifacts@1
    inputs:
      pathToPublish: '$(build.artifactstagingdirectory)'
      artifactName: 'ios_artifacts'

- job: mac_agent
  dependsOn: self_hosted_connect   
  timeoutInMinutes: 30

  pool:
    vmImage: 'macOS 10.14'
  steps:
  - script: echo 'Setting up macOS 10.14'
 
  - task: UseRubyVersion@0
    inputs:
     versionSpec: '>= 2.4'
     addToPath: true 
 
  - task: DownloadBuildArtifacts@0
    inputs:
      buildType: 'current'
      downloadType: 'single'
      artifactName: 'ios_artifacts'
      downloadPath: '$(System.ArtifactsDirectory)'  
  
  - task: Cache@2
    inputs:
       key: 'gems | "$(Agent.OS)" | my.gemspec'
       restoreKeys: | 
        gems | "$(Agent.OS)"
        gems
       path: $(BUNDLE_PATH)
    displayName: Cache gems
  
  - script:  |
     
     gem install --no-document bundler
     bundle update --bundler
     bundle install --retry=3 --jobs=4 
     gem install --no-document fastlane    
     
    workingDirectory: '$(System.ArtifactsDirectory)/ios_artifacts'
    displayName: 'installing gems and cocoapods'
 
 - script:  |
     pod deintegrate      
     gem install cocoapods     
     pod install      
     pod --version
    workingDirectory: '$(System.ArtifactsDirectory)/ios_artifacts'
    displayName: 'pod and cocoapods install '
  
  - script:  |
     echo 'Start invoking Fastfile'
     fastlane release --verbose
     echo 'Done invoking Fastfile'
    failOnStderr: false 
    workingDirectory: '$(System.ArtifactsDirectory)/ios_artifacts'
    displayName: 'run fastlane'
  
  - task: PublishBuildArtifacts@1
    inputs:
      PathtoPublish: '$(Build.ArtifactStagingDirectory)'
      ArtifactName: 'Artifacts'
      publishLocation: 'Container'
      
- job: copy_back_files_to_self_hosted_connect
  dependsOn: mac_agent 
  timeoutInMinutes: 30
  pool: Default
  steps:
    - task: DownloadBuildArtifacts@0
      inputs:
        buildType: 'current'
        downloadType: 'single'
        artifactName: 'Artifacts'
        itemPattern: | 
                **/*.ipa
                **/*manifest.plist*
        downloadPath: '$(System.ArtifactsDirectory)'
    - task: CopyFiles@2
      inputs:
        SourceFolder: '$(System.ArtifactsDirectory)'
        Contents: | 
                **/*.ipa
                **/*manifest.plist*
        TargetFolder: '$(Agent.HomeDirectory)/../${{parameters.FolderCompile}}'

Answer 1

You may consider and check caching gems as it is shown here:

variables:
  BUNDLE_PATH: $(Pipeline.Workspace)/.bundle

steps:
- task: Cache@2
  inputs:
    key: 'gems | "$(Agent.OS)" | my.gemspec'
    restoreKeys: | 
      gems | "$(Agent.OS)"
      gems
    path: $(BUNDLE_PATH)
  displayName: Cache gems

- script: bundle install

Answer 2

You can try to use Self-hosted agents instead of Microsoft-hosted agents.

• In Microsoft-hosted agents, each time you run a pipeline, you get a fresh virtual machine. The virtual machine is discarded after one use.
• In Self-hosted agents, machine-level caches and configuration persist from run to run , which can boost speed.

See Capabilities and limitations of Microsoft-hosted agents in this document. If you want to install and use a self-hosted agent, you can refer to this document.