Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........X.....XX....X...........X....................X.....] duration:
15.169613735s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:33:34 UTC)
orbiter warning: context deadline exceeded
Duration: 15.169644264s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X........X.........] duration:
58.609433588s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:33:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 58.609453227s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [......XX.X..........X.XX..XX..X.......XXX......XX......X..X.] duration:
5.755556066s trace:
https://jaeger.dev.storj.io/trace/4b9b6a438213a8d3
FAIL (2025-12-28 13:33:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ) context deadline exceeded; piecestore: (Node ID: 1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, Piece ID: 3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A) context deadline exceeded
======LOGS======
Dec 28 13:32:33.002358608 (550h3m32.873441663s) [upload=482276]: Starting upload
Dec 28 13:32:33.002447837 (550h3m32.873530902s) [upload=482276]: Done starting upload
Dec 28 13:32:33.002591150 (550h3m32.873674195s) [upload=482276]: Uploading segments...
Dec 28 13:32:33.002753169 (550h3m32.873836224s) [upload=482276]: Got next segment. Inline: false
Dec 28 13:32:33.002811530 (550h3m32.873894585s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:32:33.002819145 (550h3m32.8739022s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:32:33.247993006 (550h3m33.119076051s) [seg_pos={0 0}, upload=482276]: Begin upload segment...
Dec 28 13:32:33.248079580 (550h3m33.119162625s) [seg_pos={0 0}, upload=482276]: Done begin upload segment.
Dec 28 13:32:33.248121750 (550h3m33.119204796s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.248208536 (550h3m33.119291571s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249059396 (550h3m33.120142431s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249157222 (550h3m33.120240257s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249530353 (550h3m33.120613398s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249658547 (550h3m33.120741582s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250120958 (550h3m33.121203993s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250148721 (550h3m33.121231766s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250439063 (550h3m33.121522098s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250634445 (550h3m33.12171749s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250714578 (550h3m33.121797623s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250995353 (550h3m33.122078388s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251115211 (550h3m33.122198246s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251298089 (550h3m33.122381125s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251560509 (550h3m33.122643554s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.252067014 (550h3m33.123150059s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:34.958028252 (550h3m34.829111287s) [upload=482276]: Next returned nil segment
Dec 28 13:32:34.958047918 (550h3m34.829130994s) [upload=482276]: Waiting for error group managing segments...
Dec 28 13:32:35.250704184 (550h3m35.121787219s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252256369 (550h3m35.123339405s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252724131 (550h3m35.123807166s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.254506144 (550h3m35.125589189s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.259835472 (550h3m35.130918517s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.261372689 (550h3m35.132455724s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.262099603 (550h3m35.133182638s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.266932445 (550h3m35.13801549s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.270613434 (550h3m35.141696479s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.281706528 (550h3m35.152789563s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.286167181 (550h3m35.157250226s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.458695042 (550h3m35.329778087s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508970815 (550h3m35.38005385s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508993508 (550h3m35.380076543s) [seg_pos={0 0}, upload=482276]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:32:35.509180494 (550h3m35.380263529s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509186706 (550h3m35.380269741s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509356739 (550h3m35.380439774s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509365637 (550h3m35.380448682s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509395673 (550h3m35.380478718s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509424098 (550h3m35.380507143s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509450889 (550h3m35.380533934s) [seg_pos={0 0}, upload=482276]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:32:35.509530400 (550h3m35.380613435s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:32:35.509536793 (550h3m35.380619828s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:32:35.803906000 (550h3m35.674989045s) [upload=482276]: Done uploading segments...
Dec 28 13:32:35.803927249 (550h3m35.675010294s) [upload=482276]: Upload finished. err: <nil>
Duration: 5.755584189s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........X.....XX....X...........X....................X....] duration:
18.498413371s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:33:15 UTC)
orbiter warning: context deadline exceeded
Duration: 18.498484535s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXX] duration:
5.001472898s trace:
https://jaeger.dev.storj.io/trace/244386aadc6ed931
FAIL (2025-12-28 13:33:08 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001499849s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.......XX.X..........X.XX..XX..X.......XXX......XX......X..X] duration:
30.004700711s trace:
https://jaeger.dev.storj.io/trace/4b9b6a438213a8d3
FAIL (2025-12-28 13:33:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ) context deadline exceeded; piecestore: (Node ID: 1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, Piece ID: 3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A) context deadline exceeded
======LOGS======
Dec 28 13:32:33.002358608 (550h3m32.873441663s) [upload=482276]: Starting upload
Dec 28 13:32:33.002447837 (550h3m32.873530902s) [upload=482276]: Done starting upload
Dec 28 13:32:33.002591150 (550h3m32.873674195s) [upload=482276]: Uploading segments...
Dec 28 13:32:33.002753169 (550h3m32.873836224s) [upload=482276]: Got next segment. Inline: false
Dec 28 13:32:33.002811530 (550h3m32.873894585s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:32:33.002819145 (550h3m32.8739022s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:32:33.247993006 (550h3m33.119076051s) [seg_pos={0 0}, upload=482276]: Begin upload segment...
Dec 28 13:32:33.248079580 (550h3m33.119162625s) [seg_pos={0 0}, upload=482276]: Done begin upload segment.
Dec 28 13:32:33.248121750 (550h3m33.119204796s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.248208536 (550h3m33.119291571s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249059396 (550h3m33.120142431s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249157222 (550h3m33.120240257s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249530353 (550h3m33.120613398s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249658547 (550h3m33.120741582s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250120958 (550h3m33.121203993s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250148721 (550h3m33.121231766s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250439063 (550h3m33.121522098s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250634445 (550h3m33.12171749s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250714578 (550h3m33.121797623s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250995353 (550h3m33.122078388s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251115211 (550h3m33.122198246s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251298089 (550h3m33.122381125s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251560509 (550h3m33.122643554s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.252067014 (550h3m33.123150059s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:34.958028252 (550h3m34.829111287s) [upload=482276]: Next returned nil segment
Dec 28 13:32:34.958047918 (550h3m34.829130994s) [upload=482276]: Waiting for error group managing segments...
Dec 28 13:32:35.250704184 (550h3m35.121787219s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252256369 (550h3m35.123339405s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252724131 (550h3m35.123807166s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.254506144 (550h3m35.125589189s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.259835472 (550h3m35.130918517s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.261372689 (550h3m35.132455724s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.262099603 (550h3m35.133182638s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.266932445 (550h3m35.13801549s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.270613434 (550h3m35.141696479s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.281706528 (550h3m35.152789563s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.286167181 (550h3m35.157250226s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.458695042 (550h3m35.329778087s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508970815 (550h3m35.38005385s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508993508 (550h3m35.380076543s) [seg_pos={0 0}, upload=482276]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:32:35.509180494 (550h3m35.380263529s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509186706 (550h3m35.380269741s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509356739 (550h3m35.380439774s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509365637 (550h3m35.380448682s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509395673 (550h3m35.380478718s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509424098 (550h3m35.380507143s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509450889 (550h3m35.380533934s) [seg_pos={0 0}, upload=482276]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:32:35.509530400 (550h3m35.380613435s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:32:35.509536793 (550h3m35.380619828s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:32:35.803906000 (550h3m35.674989045s) [upload=482276]: Done uploading segments...
Dec 28 13:32:35.803927249 (550h3m35.675010294s) [upload=482276]: Upload finished. err: <nil>
Duration: 30.00474711s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...........X.....XX....X...........X....................X...] duration:
18.559892676s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:32:54 UTC)
orbiter warning: context deadline exceeded
Duration: 18.559964011s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X........X........] duration:
50.105826003s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:32:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 50.105851342s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [.XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XX] duration:
5.001584737s trace:
https://jaeger.dev.storj.io/trace/58d25a44986072cd
FAIL (2025-12-28 13:32:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001615185s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [............X.....XX....X...........X....................X..] duration:
19.085990493s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:31:42 UTC)
orbiter warning: context deadline exceeded
Duration: 19.086016613s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [..XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.X] duration:
5.001443561s trace:
https://jaeger.dev.storj.io/trace/62cf28d202af8ec4
FAIL (2025-12-28 13:31:38 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001466256s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [........XX.X..........X.XX..XX..X.......XXX......XX......X..] duration:
5.812919264s trace:
https://jaeger.dev.storj.io/trace/2b041200f1e30347
FAIL (2025-12-28 13:31:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:29:58.639531113 (550h0m58.510614158s) [upload=482242]: Starting upload
Dec 28 13:29:58.639669457 (550h0m58.510752492s) [upload=482242]: Done starting upload
Dec 28 13:29:58.639790927 (550h0m58.510873972s) [upload=482242]: Uploading segments...
Dec 28 13:29:58.640333441 (550h0m58.511416476s) [upload=482242]: Got next segment. Inline: false
Dec 28 13:29:58.640392624 (550h0m58.511475669s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:29:58.640414345 (550h0m58.51149739s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:29:58.886569264 (550h0m58.757652299s) [seg_pos={0 0}, upload=482242]: Begin upload segment...
Dec 28 13:29:58.886646741 (550h0m58.757729786s) [seg_pos={0 0}, upload=482242]: Done begin upload segment.
Dec 28 13:29:58.886672791 (550h0m58.757755836s) [noise=true , address=145.79.227.20:4439 , piece_id=JVGL5PRQX6E6BOACCZ7JK7IHQLQOC354NAIPJZUU5VQHLTZWOEIQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.887683495 (550h0m58.758766541s) [noise=true , address=145.79.227.7:4294 , piece_id=3ARUIOKRVVZ7KLDB6VVEFUURHQLEJBYUY2NOFHZLUJYB5H7GKCQA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.888301993 (550h0m58.759385039s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.888722093 (550h0m58.759805128s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889128457 (550h0m58.760211492s) [noise=true , address=145.79.227.46:4603 , piece_id=FRYHL3TVJOON766ZDAJLNH3TZ4OXRHCNHTCX3MCMFXI2CJ5QT6PQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889580348 (550h0m58.760663393s) [noise=true , address=145.79.227.13:4802 , piece_id=K6I5J4TCIBI5ZDUCFNYNSCHROK5BA6PAX3PGCNAHEXKMKXCHHFKA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889762865 (550h0m58.7608459s) [noise=true , address=45.91.116.7:4011 , piece_id=2JLRY6HV4CBLQNJSEW42GMKKP6TMDCYHLGAORXZSCFB5N2IZHXWA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.891199751 (550h0m58.762282796s) [noise=true , address=145.79.227.51:4411 , piece_id=EIKOSWZRLXOUT3MBBGIWZQM5IEV4ZG7GDXH6GKGJJCMHBHNG3BDA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.891678813 (550h0m58.762761849s) [noise=true , address=145.79.227.50:4373 , piece_id=HDAADZVLAJ257EWSXPFRF2EC3P2HB7U3HQZMFBP6LWVC4252P3DQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.892085699 (550h0m58.763168734s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.892609716 (550h0m58.763692761s) [noise=true , address=145.79.227.42:4122 , piece_id=ZMQ33HLEZCMQQXUOUVE32O7TSNMPQQTEFHRYIDZG4XT63ZQ4BP5Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.893015538 (550h0m58.764098573s) [noise=true , address=45.91.116.8:4048 , piece_id=HW2EVM4SYGTDAX7CSA5Z4A6XAHFJENHJXMLU5B7RVZE5UGH6WEVA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.893626081 (550h0m58.764709127s) [noise=true , address=145.79.227.47:4265 , piece_id=4NCUPXHSWLDYHYUVZLMATWGQT2JSRWH6LXKJ6QFT573I3OSM3AKA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.894201036 (550h0m58.765284071s) [noise=true , address=145.79.227.3:4150 , piece_id=ICMWLRFBXNHYP5WFURVKD3BJJ5Z5JSMND2HLR2K4VTQEMIGOZGRQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.894671662 (550h0m58.765754728s) [noise=true , address=45.91.116.9:4263 , piece_id=JK64MUOQ7Z63JI3VXFBEKZQKJ2ZCT4MJQ32SDIT6N2GXHWP66VUA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.896608850 (550h0m58.767691886s) [noise=true , address=145.79.227.19:4077 , piece_id=PP3X2GYF7FNGRHJ2SN7NUW5ADLEFS5KGZTUSZHSYMEFXGUJGLE5Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:30:00.517630309 (550h1m0.388713364s) [upload=482242]: Next returned nil segment
Dec 28 13:30:00.517644185 (550h1m0.38872723s) [upload=482242]: Waiting for error group managing segments...
Dec 28 13:30:00.902991830 (550h1m0.774074865s) [noise=true , address=145.79.227.42:4122 , piece_id=ZMQ33HLEZCMQQXUOUVE32O7TSNMPQQTEFHRYIDZG4XT63ZQ4BP5Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.904178460 (550h1m0.775261505s) [noise=true , address=145.79.227.47:4265 , piece_id=4NCUPXHSWLDYHYUVZLMATWGQT2JSRWH6LXKJ6QFT573I3OSM3AKA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.913000588 (550h1m0.784083633s) [noise=true , address=145.79.227.19:4077 , piece_id=PP3X2GYF7FNGRHJ2SN7NUW5ADLEFS5KGZTUSZHSYMEFXGUJGLE5Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.917507580 (550h1m0.788590615s) [noise=true , address=145.79.227.3:4150 , piece_id=ICMWLRFBXNHYP5WFURVKD3BJJ5Z5JSMND2HLR2K4VTQEMIGOZGRQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.919104200 (550h1m0.790187235s) [noise=true , address=145.79.227.51:4411 , piece_id=EIKOSWZRLXOUT3MBBGIWZQM5IEV4ZG7GDXH6GKGJJCMHBHNG3BDA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.920957218 (550h1m0.792040253s) [noise=true , address=145.79.227.13:4802 , piece_id=K6I5J4TCIBI5ZDUCFNYNSCHROK5BA6PAX3PGCNAHEXKMKXCHHFKA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.921859647 (550h1m0.792942702s) [noise=true , address=145.79.227.50:4373 , piece_id=HDAADZVLAJ257EWSXPFRF2EC3P2HB7U3HQZMFBP6LWVC4252P3DQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.923841651 (550h1m0.794924706s) [noise=true , address=145.79.227.7:4294 , piece_id=3ARUIOKRVVZ7KLDB6VVEFUURHQLEJBYUY2NOFHZLUJYB5H7GKCQA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.925186592 (550h1m0.796269637s) [noise=true , address=145.79.227.46:4603 , piece_id=FRYHL3TVJOON766ZDAJLNH3TZ4OXRHCNHTCX3MCMFXI2CJ5QT6PQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.926103869 (550h1m0.797186914s) [noise=true , address=145.79.227.20:4439 , piece_id=JVGL5PRQX6E6BOACCZ7JK7IHQLQOC354NAIPJZUU5VQHLTZWOEIQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.106114940 (550h1m0.977197975s) [noise=true , address=45.91.116.9:4263 , piece_id=JK64MUOQ7Z63JI3VXFBEKZQKJ2ZCT4MJQ32SDIT6N2GXHWP66VUA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.255964778 (550h1m1.127047823s) [noise=true , address=45.91.116.8:4048 , piece_id=HW2EVM4SYGTDAX7CSA5Z4A6XAHFJENHJXMLU5B7RVZE5UGH6WEVA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.282094312 (550h1m1.153177357s) [noise=true , address=45.91.116.7:4011 , piece_id=2JLRY6HV4CBLQNJSEW42GMKKP6TMDCYHLGAORXZSCFB5N2IZHXWA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.282119982 (550h1m1.153203027s) [seg_pos={0 0}, upload=482242]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:30:01.282360480 (550h1m1.153443525s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection
Dec 28 13:30:01.282371961 (550h1m1.153455006s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282518210 (550h1m1.153601756s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 28 13:30:01.282541183 (550h1m1.153624259s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282731226 (550h1m1.153814271s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 13:30:01.282740724 (550h1m1.153823769s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282766573 (550h1m1.153849618s) [seg_pos={0 0}, upload=482242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:30:01.283083446 (550h1m1.154166482s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:30:01.283107071 (550h1m1.154190116s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:30:01.583258308 (550h1m1.454341363s) [upload=482242]: Done uploading segments...
Dec 28 13:30:01.583286001 (550h1m1.454369056s) [upload=482242]: Upload finished. err: <nil>
Duration: 5.812963908s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X........X.......] duration:
38.388770179s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:31:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 38.388789856s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [...XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.] duration:
3.517663303s trace:
https://jaeger.dev.storj.io/trace/40de9e068db51b84
FAIL (2025-12-28 13:30:54 UTC)
orbiter warning: context deadline exceeded
Duration: 3.517689153s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.........XX.X..........X.XX..XX..X.......XXX......XX......X.] duration:
5.737905017s trace:
https://jaeger.dev.storj.io/trace/2b041200f1e30347
FAIL (2025-12-28 13:30:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:29:58.639531113 (550h0m58.510614158s) [upload=482242]: Starting upload
Dec 28 13:29:58.639669457 (550h0m58.510752492s) [upload=482242]: Done starting upload
Dec 28 13:29:58.639790927 (550h0m58.510873972s) [upload=482242]: Uploading segments...
Dec 28 13:29:58.640333441 (550h0m58.511416476s) [upload=482242]: Got next segment. Inline: false
Dec 28 13:29:58.640392624 (550h0m58.511475669s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:29:58.640414345 (550h0m58.51149739s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:29:58.886569264 (550h0m58.757652299s) [seg_pos={0 0}, upload=482242]: Begin upload segment...
Dec 28 13:29:58.886646741 (550h0m58.757729786s) [seg_pos={0 0}, upload=482242]: Done begin upload segment.
Dec 28 13:29:58.886672791 (550h0m58.757755836s) [noise=true , address=145.79.227.20:4439 , piece_id=JVGL5PRQX6E6BOACCZ7JK7IHQLQOC354NAIPJZUU5VQHLTZWOEIQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.887683495 (550h0m58.758766541s) [noise=true , address=145.79.227.7:4294 , piece_id=3ARUIOKRVVZ7KLDB6VVEFUURHQLEJBYUY2NOFHZLUJYB5H7GKCQA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.888301993 (550h0m58.759385039s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.888722093 (550h0m58.759805128s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889128457 (550h0m58.760211492s) [noise=true , address=145.79.227.46:4603 , piece_id=FRYHL3TVJOON766ZDAJLNH3TZ4OXRHCNHTCX3MCMFXI2CJ5QT6PQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889580348 (550h0m58.760663393s) [noise=true , address=145.79.227.13:4802 , piece_id=K6I5J4TCIBI5ZDUCFNYNSCHROK5BA6PAX3PGCNAHEXKMKXCHHFKA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889762865 (550h0m58.7608459s) [noise=true , address=45.91.116.7:4011 , piece_id=2JLRY6HV4CBLQNJSEW42GMKKP6TMDCYHLGAORXZSCFB5N2IZHXWA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.891199751 (550h0m58.762282796s) [noise=true , address=145.79.227.51:4411 , piece_id=EIKOSWZRLXOUT3MBBGIWZQM5IEV4ZG7GDXH6GKGJJCMHBHNG3BDA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.891678813 (550h0m58.762761849s) [noise=true , address=145.79.227.50:4373 , piece_id=HDAADZVLAJ257EWSXPFRF2EC3P2HB7U3HQZMFBP6LWVC4252P3DQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.892085699 (550h0m58.763168734s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.892609716 (550h0m58.763692761s) [noise=true , address=145.79.227.42:4122 , piece_id=ZMQ33HLEZCMQQXUOUVE32O7TSNMPQQTEFHRYIDZG4XT63ZQ4BP5Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.893015538 (550h0m58.764098573s) [noise=true , address=45.91.116.8:4048 , piece_id=HW2EVM4SYGTDAX7CSA5Z4A6XAHFJENHJXMLU5B7RVZE5UGH6WEVA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.893626081 (550h0m58.764709127s) [noise=true , address=145.79.227.47:4265 , piece_id=4NCUPXHSWLDYHYUVZLMATWGQT2JSRWH6LXKJ6QFT573I3OSM3AKA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.894201036 (550h0m58.765284071s) [noise=true , address=145.79.227.3:4150 , piece_id=ICMWLRFBXNHYP5WFURVKD3BJJ5Z5JSMND2HLR2K4VTQEMIGOZGRQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.894671662 (550h0m58.765754728s) [noise=true , address=45.91.116.9:4263 , piece_id=JK64MUOQ7Z63JI3VXFBEKZQKJ2ZCT4MJQ32SDIT6N2GXHWP66VUA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.896608850 (550h0m58.767691886s) [noise=true , address=145.79.227.19:4077 , piece_id=PP3X2GYF7FNGRHJ2SN7NUW5ADLEFS5KGZTUSZHSYMEFXGUJGLE5Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:30:00.517630309 (550h1m0.388713364s) [upload=482242]: Next returned nil segment
Dec 28 13:30:00.517644185 (550h1m0.38872723s) [upload=482242]: Waiting for error group managing segments...
Dec 28 13:30:00.902991830 (550h1m0.774074865s) [noise=true , address=145.79.227.42:4122 , piece_id=ZMQ33HLEZCMQQXUOUVE32O7TSNMPQQTEFHRYIDZG4XT63ZQ4BP5Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.904178460 (550h1m0.775261505s) [noise=true , address=145.79.227.47:4265 , piece_id=4NCUPXHSWLDYHYUVZLMATWGQT2JSRWH6LXKJ6QFT573I3OSM3AKA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.913000588 (550h1m0.784083633s) [noise=true , address=145.79.227.19:4077 , piece_id=PP3X2GYF7FNGRHJ2SN7NUW5ADLEFS5KGZTUSZHSYMEFXGUJGLE5Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.917507580 (550h1m0.788590615s) [noise=true , address=145.79.227.3:4150 , piece_id=ICMWLRFBXNHYP5WFURVKD3BJJ5Z5JSMND2HLR2K4VTQEMIGOZGRQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.919104200 (550h1m0.790187235s) [noise=true , address=145.79.227.51:4411 , piece_id=EIKOSWZRLXOUT3MBBGIWZQM5IEV4ZG7GDXH6GKGJJCMHBHNG3BDA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.920957218 (550h1m0.792040253s) [noise=true , address=145.79.227.13:4802 , piece_id=K6I5J4TCIBI5ZDUCFNYNSCHROK5BA6PAX3PGCNAHEXKMKXCHHFKA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.921859647 (550h1m0.792942702s) [noise=true , address=145.79.227.50:4373 , piece_id=HDAADZVLAJ257EWSXPFRF2EC3P2HB7U3HQZMFBP6LWVC4252P3DQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.923841651 (550h1m0.794924706s) [noise=true , address=145.79.227.7:4294 , piece_id=3ARUIOKRVVZ7KLDB6VVEFUURHQLEJBYUY2NOFHZLUJYB5H7GKCQA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.925186592 (550h1m0.796269637s) [noise=true , address=145.79.227.46:4603 , piece_id=FRYHL3TVJOON766ZDAJLNH3TZ4OXRHCNHTCX3MCMFXI2CJ5QT6PQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.926103869 (550h1m0.797186914s) [noise=true , address=145.79.227.20:4439 , piece_id=JVGL5PRQX6E6BOACCZ7JK7IHQLQOC354NAIPJZUU5VQHLTZWOEIQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.106114940 (550h1m0.977197975s) [noise=true , address=45.91.116.9:4263 , piece_id=JK64MUOQ7Z63JI3VXFBEKZQKJ2ZCT4MJQ32SDIT6N2GXHWP66VUA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.255964778 (550h1m1.127047823s) [noise=true , address=45.91.116.8:4048 , piece_id=HW2EVM4SYGTDAX7CSA5Z4A6XAHFJENHJXMLU5B7RVZE5UGH6WEVA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.282094312 (550h1m1.153177357s) [noise=true , address=45.91.116.7:4011 , piece_id=2JLRY6HV4CBLQNJSEW42GMKKP6TMDCYHLGAORXZSCFB5N2IZHXWA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.282119982 (550h1m1.153203027s) [seg_pos={0 0}, upload=482242]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:30:01.282360480 (550h1m1.153443525s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection
Dec 28 13:30:01.282371961 (550h1m1.153455006s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282518210 (550h1m1.153601756s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 28 13:30:01.282541183 (550h1m1.153624259s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282731226 (550h1m1.153814271s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 13:30:01.282740724 (550h1m1.153823769s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282766573 (550h1m1.153849618s) [seg_pos={0 0}, upload=482242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:30:01.283083446 (550h1m1.154166482s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:30:01.283107071 (550h1m1.154190116s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:30:01.583258308 (550h1m1.454341363s) [upload=482242]: Done uploading segments...
Dec 28 13:30:01.583286001 (550h1m1.454369056s) [upload=482242]: Upload finished. err: <nil>
Duration: 5.73793241s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X........X......] duration:
43.16895118s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:30:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 43.168980987s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............X.....XX....X...........X....................X.] duration:
13.925145154s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:30:35 UTC)
orbiter warning: context deadline exceeded
Duration: 13.925176634s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..........XX.X..........X.XX..XX..X.......XXX......XX......X] duration:
30.003258477s trace:
https://jaeger.dev.storj.io/trace/2b041200f1e30347
FAIL (2025-12-28 13:30:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:29:58.639531113 (550h0m58.510614158s) [upload=482242]: Starting upload
Dec 28 13:29:58.639669457 (550h0m58.510752492s) [upload=482242]: Done starting upload
Dec 28 13:29:58.639790927 (550h0m58.510873972s) [upload=482242]: Uploading segments...
Dec 28 13:29:58.640333441 (550h0m58.511416476s) [upload=482242]: Got next segment. Inline: false
Dec 28 13:29:58.640392624 (550h0m58.511475669s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:29:58.640414345 (550h0m58.51149739s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:29:58.886569264 (550h0m58.757652299s) [seg_pos={0 0}, upload=482242]: Begin upload segment...
Dec 28 13:29:58.886646741 (550h0m58.757729786s) [seg_pos={0 0}, upload=482242]: Done begin upload segment.
Dec 28 13:29:58.886672791 (550h0m58.757755836s) [noise=true , address=145.79.227.20:4439 , piece_id=JVGL5PRQX6E6BOACCZ7JK7IHQLQOC354NAIPJZUU5VQHLTZWOEIQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.887683495 (550h0m58.758766541s) [noise=true , address=145.79.227.7:4294 , piece_id=3ARUIOKRVVZ7KLDB6VVEFUURHQLEJBYUY2NOFHZLUJYB5H7GKCQA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.888301993 (550h0m58.759385039s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.888722093 (550h0m58.759805128s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889128457 (550h0m58.760211492s) [noise=true , address=145.79.227.46:4603 , piece_id=FRYHL3TVJOON766ZDAJLNH3TZ4OXRHCNHTCX3MCMFXI2CJ5QT6PQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889580348 (550h0m58.760663393s) [noise=true , address=145.79.227.13:4802 , piece_id=K6I5J4TCIBI5ZDUCFNYNSCHROK5BA6PAX3PGCNAHEXKMKXCHHFKA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889762865 (550h0m58.7608459s) [noise=true , address=45.91.116.7:4011 , piece_id=2JLRY6HV4CBLQNJSEW42GMKKP6TMDCYHLGAORXZSCFB5N2IZHXWA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.891199751 (550h0m58.762282796s) [noise=true , address=145.79.227.51:4411 , piece_id=EIKOSWZRLXOUT3MBBGIWZQM5IEV4ZG7GDXH6GKGJJCMHBHNG3BDA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.891678813 (550h0m58.762761849s) [noise=true , address=145.79.227.50:4373 , piece_id=HDAADZVLAJ257EWSXPFRF2EC3P2HB7U3HQZMFBP6LWVC4252P3DQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.892085699 (550h0m58.763168734s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.892609716 (550h0m58.763692761s) [noise=true , address=145.79.227.42:4122 , piece_id=ZMQ33HLEZCMQQXUOUVE32O7TSNMPQQTEFHRYIDZG4XT63ZQ4BP5Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.893015538 (550h0m58.764098573s) [noise=true , address=45.91.116.8:4048 , piece_id=HW2EVM4SYGTDAX7CSA5Z4A6XAHFJENHJXMLU5B7RVZE5UGH6WEVA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.893626081 (550h0m58.764709127s) [noise=true , address=145.79.227.47:4265 , piece_id=4NCUPXHSWLDYHYUVZLMATWGQT2JSRWH6LXKJ6QFT573I3OSM3AKA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.894201036 (550h0m58.765284071s) [noise=true , address=145.79.227.3:4150 , piece_id=ICMWLRFBXNHYP5WFURVKD3BJJ5Z5JSMND2HLR2K4VTQEMIGOZGRQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.894671662 (550h0m58.765754728s) [noise=true , address=45.91.116.9:4263 , piece_id=JK64MUOQ7Z63JI3VXFBEKZQKJ2ZCT4MJQ32SDIT6N2GXHWP66VUA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.896608850 (550h0m58.767691886s) [noise=true , address=145.79.227.19:4077 , piece_id=PP3X2GYF7FNGRHJ2SN7NUW5ADLEFS5KGZTUSZHSYMEFXGUJGLE5Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:30:00.517630309 (550h1m0.388713364s) [upload=482242]: Next returned nil segment
Dec 28 13:30:00.517644185 (550h1m0.38872723s) [upload=482242]: Waiting for error group managing segments...
Dec 28 13:30:00.902991830 (550h1m0.774074865s) [noise=true , address=145.79.227.42:4122 , piece_id=ZMQ33HLEZCMQQXUOUVE32O7TSNMPQQTEFHRYIDZG4XT63ZQ4BP5Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.904178460 (550h1m0.775261505s) [noise=true , address=145.79.227.47:4265 , piece_id=4NCUPXHSWLDYHYUVZLMATWGQT2JSRWH6LXKJ6QFT573I3OSM3AKA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.913000588 (550h1m0.784083633s) [noise=true , address=145.79.227.19:4077 , piece_id=PP3X2GYF7FNGRHJ2SN7NUW5ADLEFS5KGZTUSZHSYMEFXGUJGLE5Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.917507580 (550h1m0.788590615s) [noise=true , address=145.79.227.3:4150 , piece_id=ICMWLRFBXNHYP5WFURVKD3BJJ5Z5JSMND2HLR2K4VTQEMIGOZGRQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.919104200 (550h1m0.790187235s) [noise=true , address=145.79.227.51:4411 , piece_id=EIKOSWZRLXOUT3MBBGIWZQM5IEV4ZG7GDXH6GKGJJCMHBHNG3BDA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.920957218 (550h1m0.792040253s) [noise=true , address=145.79.227.13:4802 , piece_id=K6I5J4TCIBI5ZDUCFNYNSCHROK5BA6PAX3PGCNAHEXKMKXCHHFKA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.921859647 (550h1m0.792942702s) [noise=true , address=145.79.227.50:4373 , piece_id=HDAADZVLAJ257EWSXPFRF2EC3P2HB7U3HQZMFBP6LWVC4252P3DQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.923841651 (550h1m0.794924706s) [noise=true , address=145.79.227.7:4294 , piece_id=3ARUIOKRVVZ7KLDB6VVEFUURHQLEJBYUY2NOFHZLUJYB5H7GKCQA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.925186592 (550h1m0.796269637s) [noise=true , address=145.79.227.46:4603 , piece_id=FRYHL3TVJOON766ZDAJLNH3TZ4OXRHCNHTCX3MCMFXI2CJ5QT6PQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.926103869 (550h1m0.797186914s) [noise=true , address=145.79.227.20:4439 , piece_id=JVGL5PRQX6E6BOACCZ7JK7IHQLQOC354NAIPJZUU5VQHLTZWOEIQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.106114940 (550h1m0.977197975s) [noise=true , address=45.91.116.9:4263 , piece_id=JK64MUOQ7Z63JI3VXFBEKZQKJ2ZCT4MJQ32SDIT6N2GXHWP66VUA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.255964778 (550h1m1.127047823s) [noise=true , address=45.91.116.8:4048 , piece_id=HW2EVM4SYGTDAX7CSA5Z4A6XAHFJENHJXMLU5B7RVZE5UGH6WEVA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.282094312 (550h1m1.153177357s) [noise=true , address=45.91.116.7:4011 , piece_id=2JLRY6HV4CBLQNJSEW42GMKKP6TMDCYHLGAORXZSCFB5N2IZHXWA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.282119982 (550h1m1.153203027s) [seg_pos={0 0}, upload=482242]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:30:01.282360480 (550h1m1.153443525s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection
Dec 28 13:30:01.282371961 (550h1m1.153455006s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282518210 (550h1m1.153601756s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 28 13:30:01.282541183 (550h1m1.153624259s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282731226 (550h1m1.153814271s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 13:30:01.282740724 (550h1m1.153823769s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282766573 (550h1m1.153849618s) [seg_pos={0 0}, upload=482242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:30:01.283083446 (550h1m1.154166482s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:30:01.283107071 (550h1m1.154190116s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:30:01.583258308 (550h1m1.454341363s) [upload=482242]: Done uploading segments...
Dec 28 13:30:01.583286001 (550h1m1.454369056s) [upload=482242]: Upload finished. err: <nil>
Duration: 30.003281811s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [....XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX] duration:
5.001593427s trace:
https://jaeger.dev.storj.io/trace/40de9e068db51b84
FAIL (2025-12-28 13:29:56 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001617543s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..............X.....XX....X...........X....................X] duration:
30.000669297s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:29:43 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000691088s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X..........XX.X..........X.XX..XX..X.......XXX......XX......] duration:
18.10320316s trace:
https://jaeger.dev.storj.io/trace/558a27891941888a
FAIL (2025-12-28 13:29:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:23:58.187116097 (549h54m58.058199142s) [upload=482158]: Starting upload
Dec 28 13:23:58.187186300 (549h54m58.058269335s) [upload=482158]: Done starting upload
Dec 28 13:23:58.187267846 (549h54m58.058350881s) [upload=482158]: Uploading segments...
Dec 28 13:23:58.187396991 (549h54m58.058480037s) [upload=482158]: Got next segment. Inline: false
Dec 28 13:23:58.187446606 (549h54m58.058529642s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:23:58.187454050 (549h54m58.058537085s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:23:58.436176599 (549h54m58.307259654s) [seg_pos={0 0}, upload=482158]: Begin upload segment...
Dec 28 13:23:58.436232446 (549h54m58.307315481s) [seg_pos={0 0}, upload=482158]: Done begin upload segment.
Dec 28 13:23:58.436292521 (549h54m58.307375566s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436309613 (549h54m58.307392658s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436819764 (549h54m58.307902809s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437115497 (549h54m58.308198532s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437397063 (549h54m58.308480098s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437849885 (549h54m58.30893293s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438260658 (549h54m58.309343703s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438578433 (549h54m58.309661478s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.439685020 (549h54m58.310768065s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.440962393 (549h54m58.312045438s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.441043827 (549h54m58.312126862s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.442770315 (549h54m58.31385335s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.443904224 (549h54m58.314987269s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.445059715 (549h54m58.31614276s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.446384387 (549h54m58.317467422s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.448274727 (549h54m58.319357762s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:24:00.391896029 (549h55m0.262979064s) [upload=482158]: Next returned nil segment
Dec 28 13:24:00.391918041 (549h55m0.263001076s) [upload=482158]: Waiting for error group managing segments...
Dec 28 13:24:00.720818792 (549h55m0.591901827s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.749534231 (549h55m0.620617276s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.790658258 (549h55m0.661741303s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.794299080 (549h55m0.665382125s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.861613419 (549h55m0.732696454s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.895027544 (549h55m0.766110579s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.476833919 (549h55m1.347916964s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.478325670 (549h55m1.349408725s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.497602987 (549h55m1.368686032s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.659625855 (549h55m1.5307089s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.460067408 (549h55m2.331150453s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.496766669 (549h55m2.367849714s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544914590 (549h55m2.415997635s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544942112 (549h55m2.416025158s) [seg_pos={0 0}, upload=482158]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:24:02.545231112 (549h55m2.416314178s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection
Dec 28 13:24:02.545242214 (549h55m2.416325289s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545376911 (549h55m2.416459966s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545387511 (549h55m2.416470566s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545557895 (549h55m2.41664093s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545577131 (549h55m2.416660166s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545609193 (549h55m2.416692228s) [seg_pos={0 0}, upload=482158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:24:02.545707871 (549h55m2.416790916s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:24:02.545714513 (549h55m2.416797558s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:24:02.834205788 (549h55m2.705288843s) [upload=482158]: Done uploading segments...
Dec 28 13:24:02.834234483 (549h55m2.705317528s) [upload=482158]: Upload finished. err: <nil>
Duration: 18.103236614s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X........X.....] duration:
29.320704065s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:29:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 29.320722041s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [X....XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXX] duration:
5.001892707s trace:
https://jaeger.dev.storj.io/trace/29187b89bffcab12
FAIL (2025-12-28 13:28:49 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001920441s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X........X....] duration:
59.60492386s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:28:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 59.604954337s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............X.....XX....X...........X....................] duration:
24.026760635s trace:
https://jaeger.dev.storj.io/trace/61e5d25cca280779
FAIL (2025-12-28 13:28:25 UTC)
orbiter warning: context deadline exceeded
Duration: 24.026775744s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.X..........XX.X..........X.XX..XX..X.......XXX......XX.....] duration:
5.743163478s trace:
https://jaeger.dev.storj.io/trace/558a27891941888a
FAIL (2025-12-28 13:28:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:23:58.187116097 (549h54m58.058199142s) [upload=482158]: Starting upload
Dec 28 13:23:58.187186300 (549h54m58.058269335s) [upload=482158]: Done starting upload
Dec 28 13:23:58.187267846 (549h54m58.058350881s) [upload=482158]: Uploading segments...
Dec 28 13:23:58.187396991 (549h54m58.058480037s) [upload=482158]: Got next segment. Inline: false
Dec 28 13:23:58.187446606 (549h54m58.058529642s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:23:58.187454050 (549h54m58.058537085s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:23:58.436176599 (549h54m58.307259654s) [seg_pos={0 0}, upload=482158]: Begin upload segment...
Dec 28 13:23:58.436232446 (549h54m58.307315481s) [seg_pos={0 0}, upload=482158]: Done begin upload segment.
Dec 28 13:23:58.436292521 (549h54m58.307375566s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436309613 (549h54m58.307392658s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436819764 (549h54m58.307902809s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437115497 (549h54m58.308198532s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437397063 (549h54m58.308480098s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437849885 (549h54m58.30893293s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438260658 (549h54m58.309343703s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438578433 (549h54m58.309661478s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.439685020 (549h54m58.310768065s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.440962393 (549h54m58.312045438s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.441043827 (549h54m58.312126862s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.442770315 (549h54m58.31385335s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.443904224 (549h54m58.314987269s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.445059715 (549h54m58.31614276s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.446384387 (549h54m58.317467422s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.448274727 (549h54m58.319357762s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:24:00.391896029 (549h55m0.262979064s) [upload=482158]: Next returned nil segment
Dec 28 13:24:00.391918041 (549h55m0.263001076s) [upload=482158]: Waiting for error group managing segments...
Dec 28 13:24:00.720818792 (549h55m0.591901827s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.749534231 (549h55m0.620617276s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.790658258 (549h55m0.661741303s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.794299080 (549h55m0.665382125s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.861613419 (549h55m0.732696454s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.895027544 (549h55m0.766110579s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.476833919 (549h55m1.347916964s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.478325670 (549h55m1.349408725s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.497602987 (549h55m1.368686032s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.659625855 (549h55m1.5307089s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.460067408 (549h55m2.331150453s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.496766669 (549h55m2.367849714s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544914590 (549h55m2.415997635s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544942112 (549h55m2.416025158s) [seg_pos={0 0}, upload=482158]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:24:02.545231112 (549h55m2.416314178s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection
Dec 28 13:24:02.545242214 (549h55m2.416325289s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545376911 (549h55m2.416459966s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545387511 (549h55m2.416470566s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545557895 (549h55m2.41664093s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545577131 (549h55m2.416660166s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545609193 (549h55m2.416692228s) [seg_pos={0 0}, upload=482158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:24:02.545707871 (549h55m2.416790916s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:24:02.545714513 (549h55m2.416797558s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:24:02.834205788 (549h55m2.705288843s) [upload=482158]: Done uploading segments...
Dec 28 13:24:02.834234483 (549h55m2.705317528s) [upload=482158]: Upload finished. err: <nil>
Duration: 5.743188297s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................X.....XX....X...........X...................] duration:
22.85641019s trace:
https://jaeger.dev.storj.io/trace/61e5d25cca280779
FAIL (2025-12-28 13:28:01 UTC)
orbiter warning: context deadline exceeded
Duration: 22.856428024s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [.X....XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXX] duration:
5.001522023s trace:
https://jaeger.dev.storj.io/trace/793edd3f92a45a31
FAIL (2025-12-28 13:28:01 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001539285s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X.X..........XX.X..........X.XX..XX..X.......XXX......XX....] duration:
6.342188198s trace:
https://jaeger.dev.storj.io/trace/558a27891941888a
FAIL (2025-12-28 13:27:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:23:58.187116097 (549h54m58.058199142s) [upload=482158]: Starting upload
Dec 28 13:23:58.187186300 (549h54m58.058269335s) [upload=482158]: Done starting upload
Dec 28 13:23:58.187267846 (549h54m58.058350881s) [upload=482158]: Uploading segments...
Dec 28 13:23:58.187396991 (549h54m58.058480037s) [upload=482158]: Got next segment. Inline: false
Dec 28 13:23:58.187446606 (549h54m58.058529642s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:23:58.187454050 (549h54m58.058537085s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:23:58.436176599 (549h54m58.307259654s) [seg_pos={0 0}, upload=482158]: Begin upload segment...
Dec 28 13:23:58.436232446 (549h54m58.307315481s) [seg_pos={0 0}, upload=482158]: Done begin upload segment.
Dec 28 13:23:58.436292521 (549h54m58.307375566s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436309613 (549h54m58.307392658s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436819764 (549h54m58.307902809s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437115497 (549h54m58.308198532s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437397063 (549h54m58.308480098s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437849885 (549h54m58.30893293s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438260658 (549h54m58.309343703s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438578433 (549h54m58.309661478s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.439685020 (549h54m58.310768065s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.440962393 (549h54m58.312045438s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.441043827 (549h54m58.312126862s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.442770315 (549h54m58.31385335s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.443904224 (549h54m58.314987269s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.445059715 (549h54m58.31614276s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.446384387 (549h54m58.317467422s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.448274727 (549h54m58.319357762s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:24:00.391896029 (549h55m0.262979064s) [upload=482158]: Next returned nil segment
Dec 28 13:24:00.391918041 (549h55m0.263001076s) [upload=482158]: Waiting for error group managing segments...
Dec 28 13:24:00.720818792 (549h55m0.591901827s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.749534231 (549h55m0.620617276s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.790658258 (549h55m0.661741303s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.794299080 (549h55m0.665382125s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.861613419 (549h55m0.732696454s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.895027544 (549h55m0.766110579s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.476833919 (549h55m1.347916964s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.478325670 (549h55m1.349408725s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.497602987 (549h55m1.368686032s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.659625855 (549h55m1.5307089s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.460067408 (549h55m2.331150453s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.496766669 (549h55m2.367849714s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544914590 (549h55m2.415997635s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544942112 (549h55m2.416025158s) [seg_pos={0 0}, upload=482158]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:24:02.545231112 (549h55m2.416314178s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection
Dec 28 13:24:02.545242214 (549h55m2.416325289s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545376911 (549h55m2.416459966s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545387511 (549h55m2.416470566s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545557895 (549h55m2.41664093s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545577131 (549h55m2.416660166s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545609193 (549h55m2.416692228s) [seg_pos={0 0}, upload=482158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:24:02.545707871 (549h55m2.416790916s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:24:02.545714513 (549h55m2.416797558s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:24:02.834205788 (549h55m2.705288843s) [upload=482158]: Done uploading segments...
Dec 28 13:24:02.834234483 (549h55m2.705317528s) [upload=482158]: Upload finished. err: <nil>
Duration: 6.342217454s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................X.....XX....X...........X..................] duration:
29.697314172s trace:
https://jaeger.dev.storj.io/trace/61e5d25cca280779
FAIL (2025-12-28 13:27:38 UTC)
orbiter warning: context deadline exceeded
Duration: 29.697341124s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [..X....XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXX] duration:
5.000855188s trace:
https://jaeger.dev.storj.io/trace/386d8200950620a7
FAIL (2025-12-28 13:27:22 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000880317s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X........X...] duration:
31.864294146s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:27:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 31.864325064s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [...X....XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXX] duration:
5.000671098s trace:
https://jaeger.dev.storj.io/trace/10249e81a1900fe7
FAIL (2025-12-28 13:26:48 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00069299s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................X.....XX....X...........X.................] duration:
16.623406297s trace:
https://jaeger.dev.storj.io/trace/61e5d25cca280779
FAIL (2025-12-28 13:26:28 UTC)
orbiter warning: context deadline exceeded
Duration: 16.623435402s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.X.X..........XX.X..........X.XX..XX..X.......XXX......XX...] duration:
6.24762801s trace:
https://jaeger.dev.storj.io/trace/558a27891941888a
FAIL (2025-12-28 13:26:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:23:58.187116097 (549h54m58.058199142s) [upload=482158]: Starting upload
Dec 28 13:23:58.187186300 (549h54m58.058269335s) [upload=482158]: Done starting upload
Dec 28 13:23:58.187267846 (549h54m58.058350881s) [upload=482158]: Uploading segments...
Dec 28 13:23:58.187396991 (549h54m58.058480037s) [upload=482158]: Got next segment. Inline: false
Dec 28 13:23:58.187446606 (549h54m58.058529642s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:23:58.187454050 (549h54m58.058537085s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:23:58.436176599 (549h54m58.307259654s) [seg_pos={0 0}, upload=482158]: Begin upload segment...
Dec 28 13:23:58.436232446 (549h54m58.307315481s) [seg_pos={0 0}, upload=482158]: Done begin upload segment.
Dec 28 13:23:58.436292521 (549h54m58.307375566s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436309613 (549h54m58.307392658s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436819764 (549h54m58.307902809s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437115497 (549h54m58.308198532s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437397063 (549h54m58.308480098s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437849885 (549h54m58.30893293s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438260658 (549h54m58.309343703s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438578433 (549h54m58.309661478s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.439685020 (549h54m58.310768065s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.440962393 (549h54m58.312045438s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.441043827 (549h54m58.312126862s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.442770315 (549h54m58.31385335s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.443904224 (549h54m58.314987269s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.445059715 (549h54m58.31614276s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.446384387 (549h54m58.317467422s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.448274727 (549h54m58.319357762s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:24:00.391896029 (549h55m0.262979064s) [upload=482158]: Next returned nil segment
Dec 28 13:24:00.391918041 (549h55m0.263001076s) [upload=482158]: Waiting for error group managing segments...
Dec 28 13:24:00.720818792 (549h55m0.591901827s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.749534231 (549h55m0.620617276s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.790658258 (549h55m0.661741303s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.794299080 (549h55m0.665382125s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.861613419 (549h55m0.732696454s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.895027544 (549h55m0.766110579s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.476833919 (549h55m1.347916964s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.478325670 (549h55m1.349408725s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.497602987 (549h55m1.368686032s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.659625855 (549h55m1.5307089s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.460067408 (549h55m2.331150453s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.496766669 (549h55m2.367849714s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544914590 (549h55m2.415997635s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544942112 (549h55m2.416025158s) [seg_pos={0 0}, upload=482158]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:24:02.545231112 (549h55m2.416314178s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection
Dec 28 13:24:02.545242214 (549h55m2.416325289s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545376911 (549h55m2.416459966s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545387511 (549h55m2.416470566s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545557895 (549h55m2.41664093s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545577131 (549h55m2.416660166s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545609193 (549h55m2.416692228s) [seg_pos={0 0}, upload=482158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:24:02.545707871 (549h55m2.416790916s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:24:02.545714513 (549h55m2.416797558s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:24:02.834205788 (549h55m2.705288843s) [upload=482158]: Done uploading segments...
Dec 28 13:24:02.834234483 (549h55m2.705317528s) [upload=482158]: Upload finished. err: <nil>
Duration: 6.247656043s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X........X..] duration:
41.970455094s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:26:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 41.970481804s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..X.X..........XX.X..........X.XX..XX..X.......XXX......XX..] duration:
5.941401502s trace:
https://jaeger.dev.storj.io/trace/558a27891941888a
FAIL (2025-12-28 13:25:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:23:58.187116097 (549h54m58.058199142s) [upload=482158]: Starting upload
Dec 28 13:23:58.187186300 (549h54m58.058269335s) [upload=482158]: Done starting upload
Dec 28 13:23:58.187267846 (549h54m58.058350881s) [upload=482158]: Uploading segments...
Dec 28 13:23:58.187396991 (549h54m58.058480037s) [upload=482158]: Got next segment. Inline: false
Dec 28 13:23:58.187446606 (549h54m58.058529642s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:23:58.187454050 (549h54m58.058537085s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:23:58.436176599 (549h54m58.307259654s) [seg_pos={0 0}, upload=482158]: Begin upload segment...
Dec 28 13:23:58.436232446 (549h54m58.307315481s) [seg_pos={0 0}, upload=482158]: Done begin upload segment.
Dec 28 13:23:58.436292521 (549h54m58.307375566s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436309613 (549h54m58.307392658s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436819764 (549h54m58.307902809s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437115497 (549h54m58.308198532s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437397063 (549h54m58.308480098s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437849885 (549h54m58.30893293s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438260658 (549h54m58.309343703s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438578433 (549h54m58.309661478s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.439685020 (549h54m58.310768065s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.440962393 (549h54m58.312045438s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.441043827 (549h54m58.312126862s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.442770315 (549h54m58.31385335s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.443904224 (549h54m58.314987269s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.445059715 (549h54m58.31614276s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.446384387 (549h54m58.317467422s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.448274727 (549h54m58.319357762s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:24:00.391896029 (549h55m0.262979064s) [upload=482158]: Next returned nil segment
Dec 28 13:24:00.391918041 (549h55m0.263001076s) [upload=482158]: Waiting for error group managing segments...
Dec 28 13:24:00.720818792 (549h55m0.591901827s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.749534231 (549h55m0.620617276s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.790658258 (549h55m0.661741303s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.794299080 (549h55m0.665382125s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.861613419 (549h55m0.732696454s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.895027544 (549h55m0.766110579s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.476833919 (549h55m1.347916964s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.478325670 (549h55m1.349408725s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.497602987 (549h55m1.368686032s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.659625855 (549h55m1.5307089s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.460067408 (549h55m2.331150453s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.496766669 (549h55m2.367849714s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544914590 (549h55m2.415997635s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544942112 (549h55m2.416025158s) [seg_pos={0 0}, upload=482158]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:24:02.545231112 (549h55m2.416314178s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection
Dec 28 13:24:02.545242214 (549h55m2.416325289s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545376911 (549h55m2.416459966s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545387511 (549h55m2.416470566s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545557895 (549h55m2.41664093s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545577131 (549h55m2.416660166s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545609193 (549h55m2.416692228s) [seg_pos={0 0}, upload=482158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:24:02.545707871 (549h55m2.416790916s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:24:02.545714513 (549h55m2.416797558s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:24:02.834205788 (549h55m2.705288843s) [upload=482158]: Done uploading segments...
Dec 28 13:24:02.834234483 (549h55m2.705317528s) [upload=482158]: Upload finished. err: <nil>
Duration: 5.941538093s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [X...X....XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XX] duration:
5.000612888s trace:
https://jaeger.dev.storj.io/trace/3d4f402f8c23ae4c
FAIL (2025-12-28 13:25:50 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000628308s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................X.....XX....X...........X................] duration:
17.264510328s trace:
https://jaeger.dev.storj.io/trace/61e5d25cca280779
FAIL (2025-12-28 13:25:38 UTC)
orbiter warning: context deadline exceeded
Duration: 17.264534726s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X........X.] duration:
41.730682085s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:25:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 41.730696873s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [...X.X..........XX.X..........X.XX..XX..X.......XXX......XX.] duration:
6.114720279s trace:
https://jaeger.dev.storj.io/trace/558a27891941888a
FAIL (2025-12-28 13:25:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:23:58.187116097 (549h54m58.058199142s) [upload=482158]: Starting upload
Dec 28 13:23:58.187186300 (549h54m58.058269335s) [upload=482158]: Done starting upload
Dec 28 13:23:58.187267846 (549h54m58.058350881s) [upload=482158]: Uploading segments...
Dec 28 13:23:58.187396991 (549h54m58.058480037s) [upload=482158]: Got next segment. Inline: false
Dec 28 13:23:58.187446606 (549h54m58.058529642s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:23:58.187454050 (549h54m58.058537085s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:23:58.436176599 (549h54m58.307259654s) [seg_pos={0 0}, upload=482158]: Begin upload segment...
Dec 28 13:23:58.436232446 (549h54m58.307315481s) [seg_pos={0 0}, upload=482158]: Done begin upload segment.
Dec 28 13:23:58.436292521 (549h54m58.307375566s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436309613 (549h54m58.307392658s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436819764 (549h54m58.307902809s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437115497 (549h54m58.308198532s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437397063 (549h54m58.308480098s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437849885 (549h54m58.30893293s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438260658 (549h54m58.309343703s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438578433 (549h54m58.309661478s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.439685020 (549h54m58.310768065s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.440962393 (549h54m58.312045438s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.441043827 (549h54m58.312126862s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.442770315 (549h54m58.31385335s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.443904224 (549h54m58.314987269s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.445059715 (549h54m58.31614276s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.446384387 (549h54m58.317467422s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.448274727 (549h54m58.319357762s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:24:00.391896029 (549h55m0.262979064s) [upload=482158]: Next returned nil segment
Dec 28 13:24:00.391918041 (549h55m0.263001076s) [upload=482158]: Waiting for error group managing segments...
Dec 28 13:24:00.720818792 (549h55m0.591901827s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.749534231 (549h55m0.620617276s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.790658258 (549h55m0.661741303s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.794299080 (549h55m0.665382125s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.861613419 (549h55m0.732696454s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.895027544 (549h55m0.766110579s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.476833919 (549h55m1.347916964s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.478325670 (549h55m1.349408725s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.497602987 (549h55m1.368686032s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.659625855 (549h55m1.5307089s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.460067408 (549h55m2.331150453s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.496766669 (549h55m2.367849714s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544914590 (549h55m2.415997635s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544942112 (549h55m2.416025158s) [seg_pos={0 0}, upload=482158]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:24:02.545231112 (549h55m2.416314178s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection
Dec 28 13:24:02.545242214 (549h55m2.416325289s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545376911 (549h55m2.416459966s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545387511 (549h55m2.416470566s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545557895 (549h55m2.41664093s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545577131 (549h55m2.416660166s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545609193 (549h55m2.416692228s) [seg_pos={0 0}, upload=482158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:24:02.545707871 (549h55m2.416790916s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:24:02.545714513 (549h55m2.416797558s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:24:02.834205788 (549h55m2.705288843s) [upload=482158]: Done uploading segments...
Dec 28 13:24:02.834234483 (549h55m2.705317528s) [upload=482158]: Upload finished. err: <nil>
Duration: 6.114749815s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X........X] duration:
1m0.002712019s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:24:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 1m0.002725416s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [.X...X....XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.X] duration:
5.001535316s trace:
https://jaeger.dev.storj.io/trace/7785c6c6ed945562
FAIL (2025-12-28 13:24:40 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001557689s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................X.....XX....X...........X...............] duration:
17.63766987s trace:
https://jaeger.dev.storj.io/trace/61e5d25cca280779
FAIL (2025-12-28 13:24:32 UTC)
orbiter warning: context deadline exceeded
Duration: 17.637698667s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [....X.X..........XX.X..........X.XX..XX..X.......XXX......XX] duration:
30.002863511s trace:
https://jaeger.dev.storj.io/trace/558a27891941888a
FAIL (2025-12-28 13:24:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:23:58.187116097 (549h54m58.058199142s) [upload=482158]: Starting upload
Dec 28 13:23:58.187186300 (549h54m58.058269335s) [upload=482158]: Done starting upload
Dec 28 13:23:58.187267846 (549h54m58.058350881s) [upload=482158]: Uploading segments...
Dec 28 13:23:58.187396991 (549h54m58.058480037s) [upload=482158]: Got next segment. Inline: false
Dec 28 13:23:58.187446606 (549h54m58.058529642s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:23:58.187454050 (549h54m58.058537085s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:23:58.436176599 (549h54m58.307259654s) [seg_pos={0 0}, upload=482158]: Begin upload segment...
Dec 28 13:23:58.436232446 (549h54m58.307315481s) [seg_pos={0 0}, upload=482158]: Done begin upload segment.
Dec 28 13:23:58.436292521 (549h54m58.307375566s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436309613 (549h54m58.307392658s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.436819764 (549h54m58.307902809s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437115497 (549h54m58.308198532s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437397063 (549h54m58.308480098s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.437849885 (549h54m58.30893293s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438260658 (549h54m58.309343703s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.438578433 (549h54m58.309661478s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.439685020 (549h54m58.310768065s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.440962393 (549h54m58.312045438s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.441043827 (549h54m58.312126862s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.442770315 (549h54m58.31385335s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.443904224 (549h54m58.314987269s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.445059715 (549h54m58.31614276s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.446384387 (549h54m58.317467422s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:23:58.448274727 (549h54m58.319357762s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Uploading piece...
Dec 28 13:24:00.391896029 (549h55m0.262979064s) [upload=482158]: Next returned nil segment
Dec 28 13:24:00.391918041 (549h55m0.263001076s) [upload=482158]: Waiting for error group managing segments...
Dec 28 13:24:00.720818792 (549h55m0.591901827s) [noise=true , address=45.91.116.9:4261 , piece_id=UHIXGNE2ERN4EDQ4TFAUSBRSBVI7FYJTLRVDUWKJFWUCH3UVHA5Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.749534231 (549h55m0.620617276s) [noise=true , address=45.91.116.11:4118 , piece_id=TDMVRQY2JBWHD4HSVUUU3QMDOB5ET63ML5OXF4QA5SZLULZG2OPA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.790658258 (549h55m0.661741303s) [noise=true , address=45.91.116.10:4081 , piece_id=5UY7Z2QJ3E6HKPCCPTJUJCWJ7MHEISFX3AZQQWHGLRTYT7EGBMNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.794299080 (549h55m0.665382125s) [noise=true , address=45.91.116.12:4226 , piece_id=CYIBUA4Z4JVGAVOD7I4Q3AGOPPQSMK6CEPQDRZ24A5HKSGMMVZGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.861613419 (549h55m0.732696454s) [noise=true , address=45.91.116.13:4154 , piece_id=M3HYN7LMLNL7S6433WDMYGWWB54M35HCU2EL7MFTP3INOPO7HLSQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:00.895027544 (549h55m0.766110579s) [noise=true , address=45.91.116.14:4191 , piece_id=XFK6LUGVP25ARKONRCNS2XT6JIO6BX3SVLXEUTMPH7VSMRB7CRIQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.476833919 (549h55m1.347916964s) [noise=true , address=145.79.227.13:4802 , piece_id=CHERBKD7YMZABGU7QNUPBF7S2QZ77V3HPOEQPQCGSI4W3HKGNZ3Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.478325670 (549h55m1.349408725s) [noise=true , address=145.79.227.41:4087 , piece_id=U2MYMMQ7VTKKG67DT2WX74CTJTISVCLYYIBLZXDNONBHZ7P3GLLQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.497602987 (549h55m1.368686032s) [noise=true , address=145.79.227.29:4941 , piece_id=NYVCTAGZNSGQLVTJOQP6NW3PTJG7I4B7FHMHWLUVFKLUTZPR5AGA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:01.659625855 (549h55m1.5307089s) [noise=true , address=77.90.31.10:4520 , piece_id=GYQNACPGNC4ANBLU5DIDA6QDHZMTGHPGDP4T6EJJOLLY47M4L7EA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.460067408 (549h55m2.331150453s) [noise=true , address=145.79.227.39:4014 , piece_id=AZ26NO27HJWMCNYHZXHGNUNIIWHVKODUI3XSIMGDHKSWUSQVGKRQ, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.496766669 (549h55m2.367849714s) [noise=true , address=145.79.227.14:4838 , piece_id=YQROKD5AINNQZWWAENAV2BOMXF3GZKFGQ36TWUNYZETROWNUWHNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544914590 (549h55m2.415997635s) [noise=true , address=145.79.227.4:4185 , piece_id=7TOKJHP6VQ44LBDRRGVZ5R523HW4DJR6QRVPUINMRQ45TKFT4K7Q, seg_pos={0 0}, upload=482158]: Done uploading piece. err: <nil>
Dec 28 13:24:02.544942112 (549h55m2.416025158s) [seg_pos={0 0}, upload=482158]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:24:02.545231112 (549h55m2.416314178s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54844->145.79.227.10:4401: use of closed network connection
Dec 28 13:24:02.545242214 (549h55m2.416325289s) [noise=true , address=145.79.227.10:4401 , piece_id=XOF4VWB2HEBRHLPKUUNYSPRCKX6KYQXKOD2PWNRCEHLFCF4JX4KA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545376911 (549h55m2.416459966s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545387511 (549h55m2.416470566s) [noise=true , address=145.79.227.48:4302 , piece_id=5PMSTZPLO23KPMKIXEFD5W7FVDSPDUOVLRPKP4KP42LTDAJ6TXNA, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545557895 (549h55m2.41664093s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 28 13:24:02.545577131 (549h55m2.416660166s) [noise=true , address=145.79.227.33:4869 , piece_id=7GILTK6OUZ4RBDA55M55XNMCTSFX3JM4JH3EAMOGSWY2RZORFZ2A, seg_pos={0 0}, upload=482158]: Long tail cancellation due to optimal threshold
Dec 28 13:24:02.545609193 (549h55m2.416692228s) [seg_pos={0 0}, upload=482158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:24:02.545707871 (549h55m2.416790916s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:24:02.545714513 (549h55m2.416797558s) [upload=482158]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:24:02.834205788 (549h55m2.705288843s) [upload=482158]: Done uploading segments...
Dec 28 13:24:02.834234483 (549h55m2.705317528s) [upload=482158]: Upload finished. err: <nil>
Duration: 30.002887126s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................X.....XX....X...........X..............] duration:
18.904999211s trace:
https://jaeger.dev.storj.io/trace/61e5d25cca280779
FAIL (2025-12-28 13:23:50 UTC)
orbiter warning: context deadline exceeded
Duration: 18.905025852s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
45.638930779s trace:
https://jaeger.dev.storj.io/trace/6bddbaace493b6e3
FAIL (2025-12-28 13:23:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, Piece ID: YWS5BWFSCDJFLFDDEZF23X27KPGNUH2V6GKR5NM7HDY4352CA6RQ) context deadline exceeded; piecestore: (Node ID: 123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, Piece ID: G4GCY42A5RJNVWVSO5RPOQSWKKBGWN3UKFUFZ3W4CQI4KANUXDOA) context deadline exceeded; piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: I6S3PLU5NWIEP6YQWOEMAQ3RUZJUHB62UYMXG5ZYBNCOTDLI3U4Q) context deadline exceeded; piecestore: (Node ID: 1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, Piece ID: ZFRZYWYQHJ7IUFXYC43ANJAOKNETOA6DVH4P3KVXTME7AIAL372Q) context deadline exceeded; piecestore: (Node ID: 12EhEYh6rUckEz9JaSFuj52LUM3ujuYaGif4K4jYrE4r5PMsBsj, Piece ID: QR55QFDHT7VTUSU5KLZSEPYKEVHKZJW7ZZJQEHHRVXL2YJALDPSQ) context deadline exceeded; piecestore: (Node ID: 1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua, Piece ID: ICGJLDQVZEGRZ7G4OSDRUZGE3KYOXZRYHCU54535PBTHWL5FNBVQ) context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.21:4885 , piece_id=HZLGX57IBQY6QHTOIV6M25PWQZQ2255X66IYLI2ANITVXKLYPECQ, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.862390058 (549h46m44.733473103s) [noise=true , address=77.90.31.13:4636 , piece_id=G4GCY42A5RJNVWVSO5RPOQSWKKBGWN3UKFUFZ3W4CQI4KANUXDOA, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.862687394 (549h46m44.733770429s) [noise=true , address=145.79.227.21:4508 , piece_id=7UTNGH6F6XLUUKIARI6IPV2NELYPND37GNVRFUFWMODJCR4EIR7Q, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.863733396 (549h46m44.734816441s) [noise=true , address=45.91.116.14:4215 , piece_id=ICGJLDQVZEGRZ7G4OSDRUZGE3KYOXZRYHCU54535PBTHWL5FNBVQ, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.864099673 (549h46m44.735182708s) [noise=true , address=45.91.116.13:4187 , piece_id=G6PSGJPBM22ZJ6XGWIH57UT66HHBAX74H2CHB64UYY3SHN6PMHVA, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.864427758 (549h46m44.735510793s) [noise=true , address=77.90.31.18:4824 , piece_id=E66BHT5DATRFV7HD35LOM5KOCVIJDXYYKLDRIKVADW2UNQWN6XEA, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.864745633 (549h46m44.735828669s) [noise=true , address=145.79.227.40:4075 , piece_id=G2FDSMXGLXPCUNBDUSD4WLEGTYL3HF5EFWROUSW35FQZ5TNITYBA, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.865247138 (549h46m44.736330173s) [noise=true , address=145.79.227.34:4915 , piece_id=2UPMV7NUYNW5WEU6NIYPOF5EQ2JEJ3Y5HPEGBV3T2ROYJDHHS5YA, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.865617944 (549h46m44.736700979s) [noise=true , address=145.79.227.44:4212 , piece_id=RR67CDFL5BPK7UNAJRWA3GD475HKLGAWBPC2EW2WLXVOQMEBDNYA, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.866100313 (549h46m44.737183358s) [noise=true , address=145.79.227.33:4891 , piece_id=ZFRZYWYQHJ7IUFXYC43ANJAOKNETOA6DVH4P3KVXTME7AIAL372Q, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.867563950 (549h46m44.738646985s) [noise=true , address=145.79.227.2:4119 , piece_id=FEN4SDBWYHIXVOYFAFJKYYMDWUVRTHN4HM5Z6FUUYQMJW2IOV25Q, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.868390914 (549h46m44.739473949s) [noise=true , address=77.90.31.19:4869 , piece_id=I6S3PLU5NWIEP6YQWOEMAQ3RUZJUHB62UYMXG5ZYBNCOTDLI3U4Q, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.871855872 (549h46m44.742938907s) [noise=true , address=64.49.13.169:4707 , piece_id=5F2FISKBAOZ75V3EG7FRXRAZB3Y2R4AU2V5GZEX7ETXHL6MDEJLA, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.873518959 (549h46m44.744602004s) [noise=true , address=77.90.31.17:4777 , piece_id=YWS5BWFSCDJFLFDDEZF23X27KPGNUH2V6GKR5NM7HDY4352CA6RQ, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.874606770 (549h46m44.745689806s) [noise=true , address=77.90.31.11:4567 , piece_id=ZTC4I5FQT6WTPJNNXDO4UUX7QCIRJR64OCADFCYLW3CRZ2UOLI5A, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:44.877404888 (549h46m44.748487923s) [noise=true , address=145.79.227.18:4012 , piece_id=ZFGBDMTSQPUGXLGSIS5YMDCV2RLBJHX2SI2QZC2G7WWR5GC3CWUQ, seg_pos={0 0}, upload=482038]: Uploading piece...
Dec 28 13:15:46.341009020 (549h46m46.212092065s) [upload=482038]: Next returned nil segment
Dec 28 13:15:46.341025921 (549h46m46.212108966s) [upload=482038]: Waiting for error group managing segments...
Dec 28 13:15:46.722100059 (549h46m46.593183094s) [noise=true , address=145.79.227.15:4884 , piece_id=KOHRNP35SDYJU6MOXODY3HUU7CSPF5PCG4L5OZ7ME6OU4S7WRZTA, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:46.757285869 (549h46m46.628368904s) [noise=true , address=145.79.227.21:4508 , piece_id=7UTNGH6F6XLUUKIARI6IPV2NELYPND37GNVRFUFWMODJCR4EIR7Q, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:46.763640749 (549h46m46.634723785s) [noise=true , address=145.79.227.18:4012 , piece_id=ZFGBDMTSQPUGXLGSIS5YMDCV2RLBJHX2SI2QZC2G7WWR5GC3CWUQ, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:46.782687468 (549h46m46.653770503s) [noise=true , address=145.79.227.2:4119 , piece_id=FEN4SDBWYHIXVOYFAFJKYYMDWUVRTHN4HM5Z6FUUYQMJW2IOV25Q, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:46.846829635 (549h46m46.71791267s) [noise=true , address=145.79.227.34:4915 , piece_id=2UPMV7NUYNW5WEU6NIYPOF5EQ2JEJ3Y5HPEGBV3T2ROYJDHHS5YA, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:46.917400933 (549h46m46.788483968s) [noise=true , address=77.90.31.13:4636 , piece_id=G4GCY42A5RJNVWVSO5RPOQSWKKBGWN3UKFUFZ3W4CQI4KANUXDOA, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:46.979708959 (549h46m46.850791994s) [noise=true , address=145.79.227.33:4891 , piece_id=ZFRZYWYQHJ7IUFXYC43ANJAOKNETOA6DVH4P3KVXTME7AIAL372Q, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:47.021754230 (549h46m46.892837275s) [noise=true , address=45.91.116.13:4187 , piece_id=G6PSGJPBM22ZJ6XGWIH57UT66HHBAX74H2CHB64UYY3SHN6PMHVA, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:47.095310213 (549h46m46.966393248s) [noise=true , address=77.90.31.18:4824 , piece_id=E66BHT5DATRFV7HD35LOM5KOCVIJDXYYKLDRIKVADW2UNQWN6XEA, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:47.259384527 (549h46m47.130467572s) [noise=true , address=77.90.31.17:4777 , piece_id=YWS5BWFSCDJFLFDDEZF23X27KPGNUH2V6GKR5NM7HDY4352CA6RQ, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:47.604198239 (549h46m47.475281284s) [noise=true , address=77.90.31.19:4869 , piece_id=I6S3PLU5NWIEP6YQWOEMAQ3RUZJUHB62UYMXG5ZYBNCOTDLI3U4Q, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:47.611803591 (549h46m47.482886626s) [noise=true , address=45.91.116.14:4215 , piece_id=ICGJLDQVZEGRZ7G4OSDRUZGE3KYOXZRYHCU54535PBTHWL5FNBVQ, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:47.656621682 (549h46m47.527704717s) [noise=true , address=145.79.227.30:4769 , piece_id=QR55QFDHT7VTUSU5KLZSEPYKEVHKZJW7ZZJQEHHRVXL2YJALDPSQ, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:47.738821905 (549h46m47.60990494s) [noise=true , address=145.79.227.44:4212 , piece_id=RR67CDFL5BPK7UNAJRWA3GD475HKLGAWBPC2EW2WLXVOQMEBDNYA, seg_pos={0 0}, upload=482038]: Done uploading piece. err: <nil>
Dec 28 13:15:47.738845489 (549h46m47.609928524s) [seg_pos={0 0}, upload=482038]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:15:47.739057803 (549h46m47.610140838s) [noise=true , address=77.90.31.11:4567 , piece_id=ZTC4I5FQT6WTPJNNXDO4UUX7QCIRJR64OCADFCYLW3CRZ2UOLI5A, seg_pos={0 0}, upload=482038]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 28 13:15:47.739065619 (549h46m47.610148654s) [noise=true , address=77.90.31.11:4567 , piece_id=ZTC4I5FQT6WTPJNNXDO4UUX7QCIRJR64OCADFCYLW3CRZ2UOLI5A, seg_pos={0 0}, upload=482038]: Long tail cancellation due to optimal threshold
Dec 28 13:15:47.739085586 (549h46m47.610168622s) [noise=true , address=64.49.13.169:4707 , piece_id=5F2FISKBAOZ75V3EG7FRXRAZB3Y2R4AU2V5GZEX7ETXHL6MDEJLA, seg_pos={0 0}, upload=482038]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:37602->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:37602->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37602->64.49.13.169:4707: use of closed network connection
Dec 28 13:15:47.739095555 (549h46m47.6101786s) [noise=true , address=64.49.13.169:4707 , piece_id=5F2FISKBAOZ75V3EG7FRXRAZB3Y2R4AU2V5GZEX7ETXHL6MDEJLA, seg_pos={0 0}, upload=482038]: Long tail cancellation due to optimal threshold
Dec 28 13:15:47.739246273 (549h46m47.610329308s) [noise=true , address=77.90.31.21:4885 , piece_id=HZLGX57IBQY6QHTOIV6M25PWQZQ2255X66IYLI2ANITVXKLYPECQ, seg_pos={0 0}, upload=482038]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Dec 28 13:15:47.739255460 (549h46m47.610338495s) [noise=true , address=77.90.31.21:4885 , piece_id=HZLGX57IBQY6QHTOIV6M25PWQZQ2255X66IYLI2ANITVXKLYPECQ, seg_pos={0 0}, upload=482038]: Long tail cancellation due to optimal threshold
Dec 28 13:15:47.739389305 (549h46m47.61047234s) [noise=true , address=145.79.227.40:4075 , piece_id=G2FDSMXGLXPCUNBDUSD4WLEGTYL3HF5EFWROUSW35FQZ5TNITYBA, seg_pos={0 0}, upload=482038]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 28 13:15:47.739396659 (549h46m47.610479694s) [noise=true , address=145.79.227.40:4075 , piece_id=G2FDSMXGLXPCUNBDUSD4WLEGTYL3HF5EFWROUSW35FQZ5TNITYBA, seg_pos={0 0}, upload=482038]: Long tail cancellation due to optimal threshold
Dec 28 13:15:47.739412049 (549h46m47.610495084s) [seg_pos={0 0}, upload=482038]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:15:47.739474517 (549h46m47.610557562s) [upload=482038]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:15:47.739479095 (549h46m47.610562131s) [upload=482038]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:15:48.033707556 (549h46m47.904790611s) [upload=482038]: Done uploading segments...
Dec 28 13:15:48.033729076 (549h46m47.904812121s) [upload=482038]: Upload finished. err: <nil>
Duration: 45.63896261s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.....X.X..........XX.X..........X.XX..XX..X.......XXX......X] duration:
30.003635001s trace:
https://jaeger.dev.storj.io/trace/4155ba76ff7acc50
FAIL (2025-12-28 13:23:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:23:09.804328282 (549h54m9.675411317s) [upload=482148]: Starting upload
Dec 28 13:23:09.804386624 (549h54m9.675469659s) [upload=482148]: Done starting upload
Dec 28 13:23:09.804569923 (549h54m9.675652968s) [upload=482148]: Uploading segments...
Dec 28 13:23:09.804748813 (549h54m9.675831858s) [upload=482148]: Got next segment. Inline: false
Dec 28 13:23:09.804799320 (549h54m9.675882365s) [upload=482148]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:23:09.804807225 (549h54m9.67589027s) [upload=482148]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:23:10.052614140 (549h54m9.923697185s) [seg_pos={0 0}, upload=482148]: Begin upload segment...
Dec 28 13:23:10.052750229 (549h54m9.923833274s) [seg_pos={0 0}, upload=482148]: Done begin upload segment.
Dec 28 13:23:10.052840691 (549h54m9.923923726s) [noise=true , address=45.91.116.5:4298 , piece_id=DUIP2QN67S272SM6LKVHOL6EVWTFONZ7LCQDSZ5HI54ZILT72EPA, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.052854708 (549h54m9.923937743s) [noise=true , address=45.91.116.12:4227 , piece_id=ZFQQYQG5AIW4HNXK7ASAP6GHZZS6U6SN7BAI3MLTVBIJONZ2TMIQ, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.052909883 (549h54m9.923992938s) [noise=true , address=145.79.227.14:4839 , piece_id=C5OF6VJHU3WWXIZZ7C5RKRXX7RO7S2GZTSIM6HWRUZAQMC3CWU2A, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.053474678 (549h54m9.924557713s) [noise=true , address=145.79.227.13:4803 , piece_id=4FFH7YHGY32ARDQGKVV7NZFH7KIPCAJJZ4DY4KYNT7BTFYDGDB5Q, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.054204347 (549h54m9.925287392s) [noise=true , address=45.91.116.6:4976 , piece_id=B23CGGUYWGMVPBICYAHREGU27XT2JZ65JG7PLJFROTFUCD2YPVSQ, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.054796695 (549h54m9.92587974s) [noise=true , address=145.79.227.44:4193 , piece_id=YDPNSO3TWYPU4C23LAOI57GCTLUH7HKVWIYQEYFGMJLMHHIVOKYA, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.055274606 (549h54m9.926357651s) [noise=true , address=77.90.31.13:4626 , piece_id=24R6LG46WA6YWIC4MBXLXJLKNIYRMGXATOIOKODCDIBVPBRSV3EQ, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.058868729 (549h54m9.929951784s) [noise=true , address=145.79.227.36:4518 , piece_id=KDDU7FUI3OULCALXWZ6HTQPO4KZK3O6C3TYZQO6CYGP54LFMNEQQ, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.060705327 (549h54m9.931788362s) [noise=true , address=145.79.227.46:4603 , piece_id=FP5GUCJTV7YQCCRHL3TKGQB2GJEJQPUKJTBQ2M5Z7ZHJE3OFFNBQ, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.062675469 (549h54m9.933758504s) [noise=true , address=45.91.116.10:4081 , piece_id=P4FDK5TNC34MSEFLUMOFHJQPX7PPWR7GZKMZ6GYYSSDGZLGB453A, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.064298398 (549h54m9.935381443s) [noise=true , address=145.79.227.17:4970 , piece_id=ZVNSTHU5PFIVTDDHBK5K4QXBGHR3RSA76JBMMGYGM34GJPDZ2OLQ, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.065097239 (549h54m9.936180274s) [noise=true , address=145.79.227.27:4691 , piece_id=C6HM4KOAUZ7WGXS4VZIOGVRWEAOB5LVQTR7A6Y25C6GKOMO4MBRA, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.066623546 (549h54m9.937706591s) [noise=true , address=145.79.227.25:4619 , piece_id=J4DQGTCZGLRYBO2IA5TNT2ZLXJMRWXS6WVG7R6WJBT4ZFQHJ6WIA, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.067557685 (549h54m9.93864073s) [noise=true , address=77.90.31.19:4841 , piece_id=BKOEHTAEZHGQOO6EA2YXH47CT6NQNITBPQMBBEWA7B3GHDXXJSFQ, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.068383096 (549h54m9.939466141s) [noise=true , address=45.91.116.9:4262 , piece_id=DUVU7AQX7FRHNWVXTF7ZECPJDPG2FZ55DZQPNAK7JMQUDQULFCJA, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:10.069489673 (549h54m9.940572718s) [noise=true , address=145.79.227.7:4294 , piece_id=W4JMH3WIKR6IHE37VXYAQ657IBEKL7HMQMUT2TEQORAPJGDE24TA, seg_pos={0 0}, upload=482148]: Uploading piece...
Dec 28 13:23:11.673661129 (549h54m11.544744164s) [upload=482148]: Next returned nil segment
Dec 28 13:23:11.673678483 (549h54m11.544761528s) [upload=482148]: Waiting for error group managing segments...
Dec 28 13:23:12.062511694 (549h54m11.933594749s) [noise=true , address=145.79.227.27:4691 , piece_id=C6HM4KOAUZ7WGXS4VZIOGVRWEAOB5LVQTR7A6Y25C6GKOMO4MBRA, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.062704191 (549h54m11.933787236s) [noise=true , address=145.79.227.7:4294 , piece_id=W4JMH3WIKR6IHE37VXYAQ657IBEKL7HMQMUT2TEQORAPJGDE24TA, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.070830214 (549h54m11.941913259s) [noise=true , address=145.79.227.36:4518 , piece_id=KDDU7FUI3OULCALXWZ6HTQPO4KZK3O6C3TYZQO6CYGP54LFMNEQQ, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.072329268 (549h54m11.943412304s) [noise=true , address=145.79.227.17:4970 , piece_id=ZVNSTHU5PFIVTDDHBK5K4QXBGHR3RSA76JBMMGYGM34GJPDZ2OLQ, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.072998773 (549h54m11.944081818s) [noise=true , address=145.79.227.25:4619 , piece_id=J4DQGTCZGLRYBO2IA5TNT2ZLXJMRWXS6WVG7R6WJBT4ZFQHJ6WIA, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.078900592 (549h54m11.949983627s) [noise=true , address=145.79.227.14:4839 , piece_id=C5OF6VJHU3WWXIZZ7C5RKRXX7RO7S2GZTSIM6HWRUZAQMC3CWU2A, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.089451442 (549h54m11.960534487s) [noise=true , address=145.79.227.13:4803 , piece_id=4FFH7YHGY32ARDQGKVV7NZFH7KIPCAJJZ4DY4KYNT7BTFYDGDB5Q, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.089778475 (549h54m11.96086151s) [noise=true , address=145.79.227.46:4603 , piece_id=FP5GUCJTV7YQCCRHL3TKGQB2GJEJQPUKJTBQ2M5Z7ZHJE3OFFNBQ, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.290726693 (549h54m12.161809728s) [noise=true , address=45.91.116.9:4262 , piece_id=DUVU7AQX7FRHNWVXTF7ZECPJDPG2FZ55DZQPNAK7JMQUDQULFCJA, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.400450906 (549h54m12.271533941s) [noise=true , address=77.90.31.13:4626 , piece_id=24R6LG46WA6YWIC4MBXLXJLKNIYRMGXATOIOKODCDIBVPBRSV3EQ, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.406170417 (549h54m12.277253463s) [noise=true , address=45.91.116.10:4081 , piece_id=P4FDK5TNC34MSEFLUMOFHJQPX7PPWR7GZKMZ6GYYSSDGZLGB453A, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.419997065 (549h54m12.2910801s) [noise=true , address=45.91.116.12:4227 , piece_id=ZFQQYQG5AIW4HNXK7ASAP6GHZZS6U6SN7BAI3MLTVBIJONZ2TMIQ, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.464829232 (549h54m12.335912267s) [noise=true , address=45.91.116.5:4298 , piece_id=DUIP2QN67S272SM6LKVHOL6EVWTFONZ7LCQDSZ5HI54ZILT72EPA, seg_pos={0 0}, upload=482148]: Done uploading piece. err: <nil>
Dec 28 13:23:12.464848629 (549h54m12.335931674s) [seg_pos={0 0}, upload=482148]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:23:12.465115287 (549h54m12.336198332s) [noise=true , address=145.79.227.44:4193 , piece_id=YDPNSO3TWYPU4C23LAOI57GCTLUH7HKVWIYQEYFGMJLMHHIVOKYA, seg_pos={0 0}, upload=482148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:12.465124946 (549h54m12.336207981s) [noise=true , address=145.79.227.44:4193 , piece_id=YDPNSO3TWYPU4C23LAOI57GCTLUH7HKVWIYQEYFGMJLMHHIVOKYA, seg_pos={0 0}, upload=482148]: Long tail cancellation due to optimal threshold
Dec 28 13:23:12.465356296 (549h54m12.336439341s) [noise=true , address=45.91.116.6:4976 , piece_id=B23CGGUYWGMVPBICYAHREGU27XT2JZ65JG7PLJFROTFUCD2YPVSQ, seg_pos={0 0}, upload=482148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:12.465371926 (549h54m12.336454961s) [noise=true , address=45.91.116.6:4976 , piece_id=B23CGGUYWGMVPBICYAHREGU27XT2JZ65JG7PLJFROTFUCD2YPVSQ, seg_pos={0 0}, upload=482148]: Long tail cancellation due to optimal threshold
Dec 28 13:23:12.465426550 (549h54m12.336509595s) [noise=true , address=77.90.31.19:4841 , piece_id=BKOEHTAEZHGQOO6EA2YXH47CT6NQNITBPQMBBEWA7B3GHDXXJSFQ, seg_pos={0 0}, upload=482148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:12.465438432 (549h54m12.336521478s) [noise=true , address=77.90.31.19:4841 , piece_id=BKOEHTAEZHGQOO6EA2YXH47CT6NQNITBPQMBBEWA7B3GHDXXJSFQ, seg_pos={0 0}, upload=482148]: Long tail cancellation due to optimal threshold
Dec 28 13:23:12.465464653 (549h54m12.336547698s) [seg_pos={0 0}, upload=482148]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:23:12.465555505 (549h54m12.33663855s) [upload=482148]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:12.465564042 (549h54m12.336647107s) [upload=482148]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:12.753403956 (549h54m12.624486991s) [upload=482148]: Done uploading segments...
Dec 28 13:23:12.753420357 (549h54m12.624503402s) [upload=482148]: Upload finished. err: <nil>
Duration: 30.003648335s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [X.X...X....XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.] duration:
3.572988417s trace:
https://jaeger.dev.storj.io/trace/7c8d72ba491b5936
FAIL (2025-12-28 13:23:24 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.573017874s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [XX.X...X....XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X] duration:
5.001108773s trace:
https://jaeger.dev.storj.io/trace/7c8d72ba491b5936
FAIL (2025-12-28 13:22:54 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001133739s
core
uplink
us-select-1
us1-placement-20
|