Skip to content

Commit 2aaa5b8

Browse files
Anton-LatukhaAnton-Latukha
authored
CI: {caching,test,bench}: mk cache aware of package dep versions (#2532)
* CI: {caching,test,bench}: mk cache aware of package dep versions 1. CI caches 3rd patry dependency binaries. 2. That is why we do not want to trigger on all `.cabal` changes. We want to cache only when changes in `.cabal` descriptions lead to changes in versions of deps used, or in their compilation flags. `freeze` files: For example the bits of the resulting `.freeze` file: active-repositories: hackage.haskell.org:merge constraints: any.Boolean ==0.2.4, ... any.HsYAML ==0.2.1.0, HsYAML -exe, any.HsYAML-aeson ==0.2.0.1, HsYAML-aeson -exe, any.JuicyPixels ==3.3.6, JuicyPixels -mmap, any.MemoTrie ==0.6.10, MemoTrie -examples, ... any.QuickCheck ==2.14.2, QuickCheck -old-random +templatehaskell, aeson -bytestring-builder -cffi -developer -fast, ... any.aeson-pretty ==0.8.9, aeson-pretty -lib-only, bifunctors +semigroups +tagged, any.blaze-textual ==0.2.2.1, blaze-textual -developer -integer-simple +native, ... any.brittany ==0.13.1.2, brittany -brittany-dev-lib -brittany-test-perf, ... any.clock ==0.8.2, clock -llvm, any.cmdargs ==0.10.21, cmdargs +quotation -testprog, ... any.comonad ==5.0.8, comonad +containers +distributive +indexed-traversable, ... any.hashtables ==1.2.4.2, hashtables -bounds-checking -debug -detailed-profiling -portable -sse42 +unsafe-tricks, haskell-language-server -alternatenumberformat +brittany +callhierarchy +class +eval +floskell +fourmolu +haddockcomments +hlint -ignore-plugins-ghc-bounds +importlens +modulename +ormolu -pedantic +pragmas +qualifyimportednames +refineimports -rename +retrie +splice +stylishhaskell +tactic, ... any.hlint ==3.2.7, hlint -ghc-lib +gpl -hsyaml +threaded, hls-eval-plugin -pedantic, hls-graph -embed-files -pedantic -stm-stats, hls-hlint-plugin -ghc-lib -hlint33 -pedantic, ... any.integer-logarithms ==1.0.3.1, integer-logarithms -check-bounds +integer-gmp, ... any.lens ==5.0.1, lens -benchmark-uniplate -dump-splices +inlining -j +test-hunit +test-properties +test-templates +trustworthy, ... any.scientific ==0.3.7.0, scientific -bytestring-builder -integer-simple, any.semigroupoids ==5.3.6, semigroupoids +comonad +containers +contravariant +distributive +tagged +unordered-containers, any.semigroups ==0.19.2, semigroups +binary +bytestring -bytestring-builder +containers +deepseq +hashable +tagged +template-haskell +text +transformers +unordered-containers, ... any.transformers-compat ==0.6.6, transformers-compat -five +five-three -four +generic-deriving +mtl -three -two, ... any.vector ==0.12.3.1, vector +boundschecks -internalchecks -unsafechecks -wall, any.vector-algorithms ==0.8.0.4, vector-algorithms +bench +boundschecks -internalchecks -llvm +properties -unsafechecks, ... index-state: hackage.haskell.org 2021-11-29T12:28:17Z ``` You got the idea. `freeze` collects stuff for dependencies. I also checked - it solves & collects all package configuration across all subprojects in one file. * CI: {caching,test,bench}: freeze: allow to fail Before we not had this at all. Now - most cases would use it. But for example, all HLS subprojects can not solve single version of deps for GHC 9.0 at once. And depending on `.cabal` is not the right way, as it means to account of only 1 configuration of many in the project. * CI: {caching,test,bench}: fx cache keys * CI: {caching,test,bench}: freeze: output result into log * CI: caching: add workaround to allow last step to fail * CI: {caching,test,bench}: use only freeze file hash
1 parent ab8c52b commit 2aaa5b8

File tree

3 files changed

+42
-6
lines changed

3 files changed

+42
-6
lines changed

‎.github/workflows/bench.yml

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,15 @@ jobs:
7474
INDEX_STATE1=$(echo "$INDEX_STATE_ENTRY" | cut -d' ' -f2 | tr ':' '-')
7575
echo "INDEX_STATE=$INDEX_STATE1" >> $GITHUB_ENV
7676
77+
- name: Form the package list ('cabal.project.freeze')
78+
continue-on-error: true
79+
run: |
80+
cabal v2-freeze
81+
echo ''
82+
echo 'Output:'
83+
echo ''
84+
cat 'cabal.project.freeze'
85+
7786
- name: Hackage sources cache
7887
uses: actions/cache@v2
7988
env:
@@ -84,12 +93,13 @@ jobs:
8493
restore-keys: ${{ env.cache-name }}-
8594

8695
- name: Compiled deps cache
96+
id: compiled-deps
8797
uses: actions/cache@v2
8898
env:
8999
cache-name: compiled-deps
90100
with:
91101
path: ${{ steps.HaskEnvSetup.outputs.cabal-store }}
92-
key: ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project') }}
102+
key: ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project.freeze') }}
93103
restore-keys: |
94104
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-
95105
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-

‎.github/workflows/caching.yml

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,15 @@ jobs:
125125
INDEX_STATE1=$(echo "$INDEX_STATE_ENTRY" | cut -d' ' -f2 | tr ':' '-')
126126
echo "INDEX_STATE=$INDEX_STATE1" >> $GITHUB_ENV
127127
128+
- name: Form the package list ('cabal.project.freeze')
129+
continue-on-error: true
130+
run: |
131+
cabal v2-freeze
132+
echo ''
133+
echo 'Output:'
134+
echo ''
135+
cat 'cabal.project.freeze'
136+
128137
# 2021-12-02: NOTE: Cabal Hackage source tree storage does not depend on OS or GHC really,
129138
# but can depend on `base`.
130139
# But this caching is happens only inside `master` for `master` purposes of compiling the deps
@@ -135,9 +144,11 @@ jobs:
135144
env:
136145
cache-name: hackage-sources
137146
with:
138-
path: ${{ env.CABAL_PKGS_DIR }}
139-
key: ${{ env.cache-name }}-${{ env.INDEX_STATE }}
140-
restore-keys: ${{ env.cache-name }}-
147+
path: ${{ env.CABAL_PKGS_DIR }}
148+
key: ${{ env.cache-name }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project.freeze') }}
149+
restore-keys: |
150+
${{ env.cache-name }}-${{ env.INDEX_STATE }}-
151+
${{ env.cache-name }}-
141152
142153
- name: Compiled deps cache
143154
id: compiled-deps
@@ -146,7 +157,7 @@ jobs:
146157
cache-name: compiled-deps
147158
with:
148159
path: ${{ steps.HaskEnvSetup.outputs.cabal-store }}
149-
key: ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project') }}
160+
key: ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project.freeze') }}
150161
restore-keys: |
151162
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-
152163
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-
@@ -171,3 +182,8 @@ jobs:
171182
continue-on-error: true
172183
run: |
173184
cabal $cabalBuild || cabal $cabalBuild || cabal $cabalBuild
185+
186+
# Despite the `continue-on-error: true` directive - CI does not ignore the return code of the last step
187+
- name: Workaround to CI platform
188+
run: |
189+
true

‎.github/workflows/test.yml

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,15 @@ jobs:
123123
INDEX_STATE1=$(echo "$INDEX_STATE_ENTRY" | cut -d' ' -f2 | tr ':' '-')
124124
echo "INDEX_STATE=$INDEX_STATE1" >> $GITHUB_ENV
125125
126+
- name: Form the package list ('cabal.project.freeze')
127+
continue-on-error: true
128+
run: |
129+
cabal v2-freeze
130+
echo ''
131+
echo 'Output:'
132+
echo ''
133+
cat 'cabal.project.freeze'
134+
126135
- name: Hackage sources cache
127136
uses: actions/cache@v2
128137
env:
@@ -133,12 +142,13 @@ jobs:
133142
restore-keys: ${{ env.cache-name }}-
134143

135144
- name: Compiled deps cache
145+
id: compiled-deps
136146
uses: actions/cache@v2
137147
env:
138148
cache-name: compiled-deps
139149
with:
140150
path: ${{ steps.HaskEnvSetup.outputs.cabal-store }}
141-
key: ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project') }}
151+
key: ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project.freeze') }}
142152
restore-keys: |
143153
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-
144154
${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-

0 commit comments

Comments
 (0)