HDDS-10431. Merge recent commits from master (7c8160fe) to HDDS-7593 #6286

smengcl · 2024-02-27T20:28:19Z

What changes were proposed in this pull request?

Merge recent commits from master to HDDS-7593. Resolve code conflicts (mostly imports), and manually fix changes that are not solved by auto merge:

Adapt to HDDS-10226 in TestOmMetadataManager and TestOMRecoverLeaseRequest.
Adapt to HDDS-10410 in FilePerBlockStrategy.
~~Set modification time in test util createOmKeyInfo to fix the test failure of testLeaseSoftLimitForHsyncRecoverFile: cd7c790~~ This is properly fixed in master branch in HDDS-10482, which is also included in this merge.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10431

How was this patch tested?

Green CI.

…on (apache#6186)

…#5037)

Co-authored-by: Tsz-Wo Nicholas Sze <[email protected]>

… path for listKeys (apache#6195)

…e#6135)

Co-authored-by: Ritesh H Shukla <[email protected]>

… dashboard (apache#6183) Co-authored-by: tanvipenumudy <[email protected]>

…6189) Co-authored-by: ashishk <[email protected]> (cherry picked from commit 370b9d7)

…pache#5668)

…missioning datanode (apache#6185)

…espite success (apache#6217)

)

…apache#6198)

Co-authored-by: Ritesh H Shukla <[email protected]>

apache#6322)

…#6268)

…pache#6328)

jojochuang · 2024-03-05T18:11:16Z

Cherrypicking HDDS-9843. Ozone client high memory (heap) utilization on top of
HDDS-9751. [hsync] Make Putblock performance acceptable - DataNode side (#5662)
failed that test. Cherrypicking HDDS-9843 on the commit before does not fail. HDFS-9751 is what caused the problem.

smengcl · 2024-03-05T18:57:31Z

During CI debugging, I've noticed that because unit job depends on basic but not on build (plus the fact that build takes longer than basic), a race can unintentionally cause the unit job to use an old cached ozone build to run the actual test. This caused my change to print extra debugging log in my branch to not take effect.

My question: Should we make build a dependency of unit? Or is there a reason not to do so? Posted PR #6333

cc @adoroszlai

… branch. (apache#6310)

apache#6329)

jojochuang · 2024-03-06T18:50:22Z

Apart from HDDS-10471, another problem is TestOMRecoverLeaseRequest#testLeaseSoftLimitForHsyncRecoverFile fails consistently. It looks like a regression in the master branch where modification time of an open key is always zero.

smengcl · 2024-03-06T19:43:34Z

Just brought 3 new commits including the Direct ByteBuffer crash fix HDDS-10471 into the dev branch:

be5ffc29e2 HDDS-10471. [hsync] MockDatanodeStorage.writeChunk should make a copy of byte string. (#6334)
f244a67037 HDDS-10252. [hsync] Revisit configuration keys for incremental chunk list after HDDS-9884 (#6289)
1cf9e95eef HDDS-10427. Retry read wait based on policy. (#6292)

jojochuang

+1

please make sure the git history is not squashed together. Otherwise we can merge it. Thanks!

…ation time (apache#6343)

jojochuang · 2024-03-07T21:43:32Z

LGTM

smengcl · 2024-03-07T21:49:00Z

Thanks @jojochuang for reviewing this. I will push from local in a minute.

smengcl · 2024-03-07T21:50:32Z

Done.

Also thanks @jojochuang for all the help on resolving the test failures in this one!

kerneltime and others added 30 commits February 8, 2024 06:14

HDDS-10229. Fixes for Grafana dashboards (apache#6120)

039dea9

HDDS-10296. Orphan blocks during overwrite of key. (apache#6180)

3d8365f

HDDS-10226. Refactor OMRequestTestUtils.createOmKeyInfo (apache#6184)

06399b5

HDDS-10291. Set simple properties in MiniOzoneCluster via Configurati…

e93e781

…on (apache#6186)

HDDS-10328. Support cross realm Kerberos out of box. (apache#6192)

5715aee

HDDS-10322. Make VolumeArgs immutable (apache#6193)

601fd41

HDDS-8627. Recon - API for Count of deletePending directories (apache…

c1efa33

…#5037)

HDDS-10333. RocksDB logger not closed (apache#6200)

15b62de

HDDS-9843. Ozone client high memory (heap) utilization (apache#6153)

75df6c1

Co-authored-by: Tsz-Wo Nicholas Sze <[email protected]>

HDDS-10319. Also consider bucket layout deciding whether to normalize…

d3e2e59

… path for listKeys (apache#6195)

HDDS-10262. Encapsulate SnapshotCache inside OmSnapshotManager (apach…

2f2234c

…e#6135)

HDDS-10340. Skip ci on dashboard updates (apache#6203)

af25a48

Co-authored-by: Ritesh H Shukla <[email protected]>

HDDS-10318. Add OM client protocol metrics to Ozone - ListKey Metrics…

47ef84c

… dashboard (apache#6183) Co-authored-by: tanvipenumudy <[email protected]>

HDDS-10256. Retry block allocation when SCM is in safe mode. (apache#…

7c79246

…6189) Co-authored-by: ashishk <[email protected]> (cherry picked from commit 370b9d7)

HDDS-10250. Use SnapshotId as key in SnapshotCache (apache#6139)

c35e99f

HDDS-10343. Remove dependency on jsr305 (apache#6208)

ffd8221

HDDS-10218. Speed up TestSstFilteringService (apache#6196)

cd00691

HDDS-10325. Make BucketArgs immutable (apache#6205)

45c853c

HDDS-10344. Schedule dependabot for weekend (apache#6209)

c289c67

HDDS-10347. Bump jacoco to 0.8.11 (apache#6214)

68662a7

HDDS-10356. Bump exec-maven-plugin to 3.1.1 (apache#6215)

bacb184

HDDS-9680. Use md5 hash of multipart object part's content as ETag (a…

7370676

…pache#5668)

HDDS-9738. Display startTime, pipeline and container counts for decom…

3c4683e

…missioning datanode (apache#6185)

HDDS-10359. Recursively deleting volume with OBS bucket shows error d…

44adf80

…espite success (apache#6217)

HDDS-10368. Bump guice to 6.0.0 (apache#6212)

3876852

HDDS-10369. Set Times API doesn't work with linked buckets. (apache#6220

78fc781

)

HDDS-10301. Recon - Fold the pipeline info for a DN on Datanode page. (…

6194d42

…apache#6198)

HDDS-10288. Checksum to support direct buffers (apache#6162)

2348784

HDDS-10339. Add S3 API level dashboard (apache#6202)

cce2f96

Co-authored-by: Ritesh H Shukla <[email protected]>

HDDS-10391. Bump joda-time to 2.12.7 (apache#6230)

e0f3ae1

Tejaskriya and others added 3 commits March 5, 2024 11:34

HDDS-10458. Mention ozone admin datanode status decommission in docs (

8059213

apache#6322)

HDDS-10412. Prefix ACL check needs to resolve the bucket link (apache…

b69674c

…#6268)

HDDS-10424. Improve error message for prefix without trailing slash (a…

f16b1af

…pache#6328)

HDDS-10457. Remove dependency commons-pool2 (apache#6317)

9a6ece2

smengcl mentioned this pull request Mar 5, 2024

HDDS-10468. CI: Make unit depend on build #6333

Closed

HDDS-10450. Add GitHub actions labeler for the reconciliation feature…

4243721

… branch. (apache#6310)

jojochuang mentioned this pull request Mar 5, 2024

HDDS-10471. [hsync] MockDatanodeStorage.writeChunk should make a copy of byte string. #6334

Merged

adoroszlai and others added 3 commits March 6, 2024 13:13

HDDS-10467. Reduce metrics visibility (apache#6332)

87d8d61

HDDS-10384. RPC client reusing thread resources. (apache#6326)

2f05353

HDDS-10460. Refine audit logging for bucket property update operations (

418528a

apache#6329)

adoroszlai and others added 2 commits March 6, 2024 20:13

HDDS-10472. Audit log should include EC replication config (apache#6338)

309e459

Merge remote-tracking branch 'asf/HDDS-7593' into HDDS-7593

1d81c32

HDDS-9343. Shift sortDatanodes logic to OM (apache#5391)

140c5de

smengcl force-pushed the HDDS-7593 branch from 25f42bc to cd7c790 Compare March 6, 2024 22:54

jojochuang approved these changes Mar 6, 2024

View reviewed changes

smengcl force-pushed the HDDS-7593 branch from cd7c790 to 693bbe8 Compare March 6, 2024 23:44

adoroszlai and others added 2 commits March 7, 2024 06:13

HDDS-9343. (addendum) Shift sortDatanodes logic to OM (apache#5391)

a145dd5

HDDS-10482. OMRequestTestUtils.createOmKeyInfo should set key modific…

7c8160f

…ation time (apache#6343)

smengcl force-pushed the HDDS-7593 branch from 693bbe8 to cd7c790 Compare March 7, 2024 18:35

Merge remote-tracking branch 'asf/master' into HDDS-7593

3cadb24

smengcl force-pushed the HDDS-7593 branch from cd7c790 to 3cadb24 Compare March 7, 2024 18:39

smengcl changed the title ~~HDDS-10431. Merge recent commits from master (54548aa7) to HDDS-7593~~ HDDS-10431. Merge recent commits from master (7c8160fe) to HDDS-7593 Mar 7, 2024

smengcl merged commit 3cadb24 into apache:HDDS-7593 Mar 7, 2024
41 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-10431. Merge recent commits from master (7c8160fe) to HDDS-7593 #6286

HDDS-10431. Merge recent commits from master (7c8160fe) to HDDS-7593 #6286

smengcl commented Feb 27, 2024 •

edited

Loading

jojochuang commented Mar 5, 2024

smengcl commented Mar 5, 2024 •

edited

Loading

jojochuang commented Mar 6, 2024

smengcl commented Mar 6, 2024

jojochuang left a comment

jojochuang commented Mar 7, 2024

smengcl commented Mar 7, 2024

smengcl commented Mar 7, 2024

HDDS-10431. Merge recent commits from master (7c8160fe) to HDDS-7593 #6286

HDDS-10431. Merge recent commits from master (7c8160fe) to HDDS-7593 #6286

Conversation

smengcl commented Feb 27, 2024 • edited Loading

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

jojochuang commented Mar 5, 2024

smengcl commented Mar 5, 2024 • edited Loading

jojochuang commented Mar 6, 2024

smengcl commented Mar 6, 2024

jojochuang left a comment

Choose a reason for hiding this comment

jojochuang commented Mar 7, 2024

smengcl commented Mar 7, 2024

smengcl commented Mar 7, 2024

smengcl commented Feb 27, 2024 •

edited

Loading

smengcl commented Mar 5, 2024 •

edited

Loading