Summary:
- parse folly getdeps manifests with bare package entries so fallback prefetching actually runs
- validate and remove bad cached/downloaded archives before trying fallback mirrors
- download through temporary files and include libiberty in the GNU toolchain fallback set
Context:
Nightly test failed with dependency download failure in folly.
```
Assessing autoconf...
Download with https://ftpmirror.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz -> /tmp/fbcode_builder_getdeps-Z__wZrocksdbZrocksdbZthird-partyZfollyZbuildZfbcode_builder-root/downloads/autoconf-autoconf-2.69.tar.gz ...
[Complete in 136.616022 seconds]
raise Exception(
Exception: https://ftpmirror.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz: expected sha256 954bd69b391edc12d6a4a51a2dd1476543da5c6bbf05a95b59dc0dd6fd4c2969 but got e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
make: *** [folly.mk:152: build_folly] Error 1
##[error]Process completed with exit code 2.
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14763
Test Plan:
1. Connection failure:
- Forced first mirror to http://127.0.0.1:1/...
- It logged connection refused.
- It then tried https://mirrors.kernel.org/gnu/...
- Download succeeded, size 1927468, SHA matched 954bd69b...
2. Empty file / bad hash:
- Ran a local HTTP server returning a zero-byte autoconf-2.69.tar.gz
- Script logged mismatch with actual=e3b0c442... size=0
- It removed the bad download and fell back to mirrors.kernel.org
- Download succeeded with the expected SHA.
3. Existing zero-byte cache:
- Seeded cache with an empty tarball.
- Script removed invalid cache and downloaded a verified copy.
Reviewed By: mszeszko-meta
Differential Revision: D105859558
Pulled By: xingbowang
fbshipit-source-id: ff1f20f87debad561610271ce99b8b8de2d4264f
Summary:
In my last version bump, I forgot that the next release would be a major release. We can fix that now ahead of release cut.
I'm also updating folly now because I have experience resolving folly issues. Folly commit e04860553 changed libevent to build as static-only so required a change in our build.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14357
Test Plan: CI
Reviewed By: xingbowang
Differential Revision: D93797666
Pulled By: pdillinger
fbshipit-source-id: 22179da900f9dc6c5544163071079a4701c7c663
Summary:
Update HISTORY, version number, format compatible test, and folly version
folly build now depends on libaio
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14259
Test Plan: CI
Reviewed By: anand1976
Differential Revision: D91356493
Pulled By: pdillinger
fbshipit-source-id: 9d85960c647758d5cb33e3910e714e2f7785fd06
Summary:
Folly download dependencies directly from external source. Sometimes, this could fail due to external website instability. To solve this, we added github cache to cache the dependencies. We also added a python script to try different sources during download to reduce the chance of failure.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14226
Test Plan: github CI
Reviewed By: krhancoc, archang19
Differential Revision: D90343051
Pulled By: xingbowang
fbshipit-source-id: 3faad6aaa6c1bfd361b9e405c298856cd64bf457
Summary:
* Fix nightly build-linux-cmake-with-folly-lite-no-test for real this time
with correct include directory. (CMakeLists.txt)
* Add test runs to that build (and rename)
* Improve folly build caching with a folly.mk file with most of the relevant
parts of Makefile that contribute to the checkout_folly and
build_folly builds. This reduces the risk of false passing of CI job with
cache folly build. This caching is still only for folly debug builds, (which
is probably OK with just a single nightly build relying on release folly
build, which also serves as a rough canary against false passing
because of caching).
* Use `make VERBOSE=1` after cmake calls for detailed output
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14099
Test Plan:
temporary CI change to put the relevant parts in pr-jobs,
then back to homes including in nightly
Reviewed By: mszeszko-meta
Differential Revision: D86243363
Pulled By: pdillinger
fbshipit-source-id: f7975fa190ef45195c6d0b74417f7886e551516a
xingbowang