Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.......X......X.........] duration: 5.873606707s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:34:36 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.873638869s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X........X........................X..........] duration: 3.689491797s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:34:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.689523528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.03918086s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:34:26 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.039212611s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.967078344s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:34:10 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.967104022s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.......X......X........] duration: 6.285323418s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:33:47 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.28536167s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.064501975s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:33:40 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.064534958s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................X........X........................X.........] duration: 3.847336142s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:33:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.847373683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X....XX..........] duration: 5.787684046s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:33:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.787713582s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.992578199s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:33:19 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 1.992607697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.......X......X.......] duration: 5.897197331s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:33:12 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.897230264s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X....XX.........] duration: 5.782583833s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:33:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.782615534s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.04469456s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:33:10 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.044722162s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.......X......X......] duration: 5.971616128s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:32:35 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.971648901s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.304523565s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:32:33 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.304551388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.061211833s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:32:33 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.061252149s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................X........X........................X........] duration: 5.001263878s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:32:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001291791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X....XX........] duration: 5.842104272s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:32:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.842133408s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X........X........................X.......] duration: 3.564123399s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:32:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.564210606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.967062953s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:31:32 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.967094473s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X....XX.......] duration: 5.72565032s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:31:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.725678654s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.0443111s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:31:25 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.044343461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X.......X......X.....] duration: 5.826681372s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:31:24 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.826711701s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X........X........................X......] duration: 3.46133786s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:31:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46136291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X....XX......] duration: 5.834284734s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:31:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.834313478s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X........X........................X.....] duration: 3.465395345s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:30:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.465422988s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.......X......X....] duration: 14.869974249s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:30:49 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 14.870008535s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X....XX.....] duration: 6.597594575s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:30:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 6.597620515s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002337053s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:30:47 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 30.002363413s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 10.849137057s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:30:41 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 10.849172064s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.678176064s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:29:47 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.678209236s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X........X........................X....] duration: 3.490170703s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:29:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490229304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X.......X......X...] duration: 5.914232115s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:29:42 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.914249589s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X....XX....] duration: 5.780323886s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:29:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.78035249s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X........X........................X...] duration: 3.652561689s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:29:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.652589039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.968824282s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:29:04 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.968856583s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X....XX...] duration: 5.794095136s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:28:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.794114653s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.947666605s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:28:36 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.947687303s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X........X........................X..] duration: 3.360467761s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:28:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.360514921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.......X......X..] duration: 5.889502132s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:28:32 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.889555954s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X....XX..] duration: 5.739228928s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:28:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.739281337s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.......X......X.] duration: 6.391110656s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:28:03 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.391135695s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X....XX.] duration: 7.233047103s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:27:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 7.233068713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 11.158142244s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:27:44 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 11.158171391s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X.......X......X] duration: 30.005379961s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:27:42 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.005520388s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X....XX] duration: 30.003710475s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:27:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 30.003732706s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................X........X........................X.] duration: 3.463112115s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:27:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.463138044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 21.661006213s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:27:12 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 21.661040669s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X....X] duration: 30.004719138s trace: https://jaeger.dev.storj.io/trace/1944bba6bcc46f4a FAIL (2025-11-29 15:27:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:26:41.481896822 (487h22m40.033367283s) [upload=429160]: Starting upload Nov 29 15:26:41.481955112 (487h22m40.033425563s) [upload=429160]: Done starting upload Nov 29 15:26:41.482182997 (487h22m40.033653478s) [upload=429160]: Uploading segments... Nov 29 15:26:41.482310479 (487h22m40.03378094s) [upload=429160]: Got next segment. Inline: false Nov 29 15:26:41.482371305 (487h22m40.033841766s) [upload=429160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:26:41.482377216 (487h22m40.033847677s) [upload=429160]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:26:41.729169431 (487h22m40.280639912s) [seg_pos={0 0}, upload=429160]: Begin upload segment... Nov 29 15:26:41.729234905 (487h22m40.280705376s) [seg_pos={0 0}, upload=429160]: Done begin upload segment. Nov 29 15:26:41.729266475 (487h22m40.280736936s) [noise=true , address=145.79.227.16:4910 , piece_id=VBNI77MAZBTCGFGA37JQUZYQAHL7DCOSBESGPUO5XLJYQQRYYQYA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.729275624 (487h22m40.280746085s) [noise=true , address=45.91.116.9:4262 , piece_id=DLO4CTRY2ZLVM6SLOQ6STR6YC4XW2544EY33BEPGRJSTZUF63THQ, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.729794772 (487h22m40.281265233s) [noise=true , address=145.79.227.21:4475 , piece_id=TSCDHYLYYIR2XHHINAVNT4HF4RQGQV2P6PVCAMXMLIDJMBZ2VQ3Q, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.730079715 (487h22m40.281550176s) [noise=true , address=145.79.227.38:4977 , piece_id=WX2SKHCZOLXK7Y4KJ66U52UDI3GJ2FXU2MBH4WZOHGU5KUARNGUA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.730258976 (487h22m40.281729437s) [noise=true , address=145.79.227.27:4691 , piece_id=NIGF67DRPFG753TH3FQZOSIM3N45KO3B4PDZCNMIOVAYSIZQWKRA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.730408450 (487h22m40.281878911s) [noise=true , address=145.79.227.8:4329 , piece_id=KOZKMMNAYYIPE7FBQRC7H3576SWFW3MZJ5PWCC22WTNPYZOSDFUA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.730652805 (487h22m40.282123256s) [noise=true , address=145.79.227.10:4401 , piece_id=RBUUL3ZYHRRJHTWCGAHBKPOSC2AOZBRAVFYUX46524UDI3BKADHQ, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.730688072 (487h22m40.282158524s) [noise=true , address=145.79.227.44:4194 , piece_id=2FBWCEC2UVGXHMLZJXZBQSPF6NVYOYDBDLLMDFTTBVDE3KRJUALQ, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.730967104 (487h22m40.282437565s) [noise=true , address=145.79.227.20:4439 , piece_id=DSO7HWXUPLYLK5F4YLS7VS5T3U625UJCRRCLR6FLCBY3NHDBQ76A, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.731079858 (487h22m40.282550319s) [noise=true , address=145.79.227.41:4085 , piece_id=AQ2A7OH6LGHOFSBTKXBSNYJLJIPD64DKUBM4KLMXEKFVPHADY7NA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.731240274 (487h22m40.282710725s) [noise=true , address=145.79.227.23:4546 , piece_id=COJF3LRRCB5LEQQLU2U5JSLEKABOJ5AY2T2OTFOIWLYRLXHCW6IA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.731504167 (487h22m40.282974628s) [noise=true , address=145.79.227.4:4185 , piece_id=OFSBQCOFCYXCCROBNWGLZG4VO5HMRCDUM5PXMAQWOKRWHUVP6FJQ, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.731521680 (487h22m40.282992131s) [noise=true , address=145.79.227.19:4079 , piece_id=3Q2LHIVJOHVK5WYGDN5GOO77XVAESHSIUDRGA2XNVA3OVV4V6ASQ, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.731979351 (487h22m40.283449812s) [noise=true , address=145.79.227.15:4876 , piece_id=3E3KI336752SN6U4WS3EXSRVZ4I2HSMXIPPYTX2OS4CTP66TWAWQ, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.732070886 (487h22m40.283541347s) [noise=true , address=45.91.116.5:4299 , piece_id=SXOKIMBIXVYQTTBJV4PMXEHLLSQEBWQGEPU6JLHYRJKZZSEHZZUA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.732458414 (487h22m40.283928905s) [noise=true , address=145.79.227.43:4159 , piece_id=XZUVACHL2QYJS2RUSH5SFLOBFCJL27BGL5BQZV5YNQHN4AAKTRGA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:43.532207571 (487h22m42.083678032s) [upload=429160]: Next returned nil segment Nov 29 15:26:43.532226867 (487h22m42.083697328s) [upload=429160]: Waiting for error group managing segments... Nov 29 15:26:43.836379197 (487h22m42.387849648s) [noise=true , address=45.91.116.5:4299 , piece_id=SXOKIMBIXVYQTTBJV4PMXEHLLSQEBWQGEPU6JLHYRJKZZSEHZZUA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:43.874486060 (487h22m42.425956521s) [noise=true , address=45.91.116.9:4262 , piece_id=DLO4CTRY2ZLVM6SLOQ6STR6YC4XW2544EY33BEPGRJSTZUF63THQ, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:45.758097878 (487h22m44.309568339s) [noise=true , address=145.79.227.19:4079 , piece_id=3Q2LHIVJOHVK5WYGDN5GOO77XVAESHSIUDRGA2XNVA3OVV4V6ASQ, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:45.759427410 (487h22m44.310897871s) [noise=true , address=145.79.227.20:4439 , piece_id=DSO7HWXUPLYLK5F4YLS7VS5T3U625UJCRRCLR6FLCBY3NHDBQ76A, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:45.812204188 (487h22m44.363674649s) [noise=true , address=145.79.227.27:4691 , piece_id=NIGF67DRPFG753TH3FQZOSIM3N45KO3B4PDZCNMIOVAYSIZQWKRA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:45.824856100 (487h22m44.376326561s) [noise=true , address=145.79.227.21:4475 , piece_id=TSCDHYLYYIR2XHHINAVNT4HF4RQGQV2P6PVCAMXMLIDJMBZ2VQ3Q, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:45.896769929 (487h22m44.44824039s) [noise=true , address=145.79.227.16:4910 , piece_id=VBNI77MAZBTCGFGA37JQUZYQAHL7DCOSBESGPUO5XLJYQQRYYQYA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:47.596076674 (487h22m46.147547125s) [noise=true , address=145.79.227.23:4546 , piece_id=COJF3LRRCB5LEQQLU2U5JSLEKABOJ5AY2T2OTFOIWLYRLXHCW6IA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:48.569567674 (487h22m47.121038135s) [noise=true , address=145.79.227.43:4159 , piece_id=XZUVACHL2QYJS2RUSH5SFLOBFCJL27BGL5BQZV5YNQHN4AAKTRGA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:48.695021133 (487h22m47.246491594s) [noise=true , address=145.79.227.41:4085 , piece_id=AQ2A7OH6LGHOFSBTKXBSNYJLJIPD64DKUBM4KLMXEKFVPHADY7NA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:49.035503100 (487h22m47.586973551s) [noise=true , address=145.79.227.44:4194 , piece_id=2FBWCEC2UVGXHMLZJXZBQSPF6NVYOYDBDLLMDFTTBVDE3KRJUALQ, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:50.485461443 (487h22m49.036931905s) [noise=true , address=145.79.227.4:4185 , piece_id=OFSBQCOFCYXCCROBNWGLZG4VO5HMRCDUM5PXMAQWOKRWHUVP6FJQ, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:51.117998887 (487h22m49.669469358s) [noise=true , address=145.79.227.15:4876 , piece_id=3E3KI336752SN6U4WS3EXSRVZ4I2HSMXIPPYTX2OS4CTP66TWAWQ, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:51.118022191 (487h22m49.66953834s) [seg_pos={0 0}, upload=429160]: Segment reached optimal threshold of 13 pieces. Nov 29 15:26:51.118374071 (487h22m49.669844532s) [noise=true , address=145.79.227.10:4401 , piece_id=RBUUL3ZYHRRJHTWCGAHBKPOSC2AOZBRAVFYUX46524UDI3BKADHQ, seg_pos={0 0}, upload=429160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Nov 29 15:26:51.118385964 (487h22m49.669856425s) [noise=true , address=145.79.227.10:4401 , piece_id=RBUUL3ZYHRRJHTWCGAHBKPOSC2AOZBRAVFYUX46524UDI3BKADHQ, seg_pos={0 0}, upload=429160]: Long tail cancellation due to optimal threshold Nov 29 15:26:51.118573571 (487h22m49.670044032s) [noise=true , address=145.79.227.8:4329 , piece_id=KOZKMMNAYYIPE7FBQRC7H3576SWFW3MZJ5PWCC22WTNPYZOSDFUA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Nov 29 15:26:51.118595002 (487h22m49.670065463s) [noise=true , address=145.79.227.8:4329 , piece_id=KOZKMMNAYYIPE7FBQRC7H3576SWFW3MZJ5PWCC22WTNPYZOSDFUA, seg_pos={0 0}, upload=429160]: Long tail cancellation due to optimal threshold Nov 29 15:26:51.118895474 (487h22m49.670365945s) [noise=true , address=145.79.227.38:4977 , piece_id=WX2SKHCZOLXK7Y4KJ66U52UDI3GJ2FXU2MBH4WZOHGU5KUARNGUA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Nov 29 15:26:51.118906756 (487h22m49.670377217s) [noise=true , address=145.79.227.38:4977 , piece_id=WX2SKHCZOLXK7Y4KJ66U52UDI3GJ2FXU2MBH4WZOHGU5KUARNGUA, seg_pos={0 0}, upload=429160]: Long tail cancellation due to optimal threshold Nov 29 15:26:51.118941863 (487h22m49.670412324s) [seg_pos={0 0}, upload=429160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:26:51.119051942 (487h22m49.670522403s) [upload=429160]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:26:51.119076790 (487h22m49.670547251s) [upload=429160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:26:51.833210599 (487h22m50.38468105s) [upload=429160]: Done uploading segments... Nov 29 15:26:51.833226039 (487h22m50.38469649s) [upload=429160]: Upload finished. err: <nil> Duration: 30.004741461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.......X......] duration: 10.495927353s trace: https://jaeger.dev.storj.io/trace/3c12e61589dba0b7 FAIL (2025-11-29 15:26:56 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 34) Duration: 10.495970846s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X........X........................X] duration: 5.001330736s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:26:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001369852s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)