Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X........X........................X..] duration: 3.360467761s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:28:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.360514921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.......X......X..] duration: 5.889502132s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:28:32 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.889555954s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X....XX..] duration: 5.739228928s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:28:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.739281337s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.......X......X.] duration: 6.391110656s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:28:03 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.391135695s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X....XX.] duration: 7.233047103s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:27:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 7.233068713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 11.158142244s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:27:44 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 11.158171391s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X.......X......X] duration: 30.005379961s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:27:42 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.005520388s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X....XX] duration: 30.003710475s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:27:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 30.003732706s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................X........X........................X.] duration: 3.463112115s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:27:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.463138044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 21.661006213s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:27:12 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 21.661040669s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X....X] duration: 30.004719138s trace: https://jaeger.dev.storj.io/trace/1944bba6bcc46f4a FAIL (2025-11-29 15:27:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:26:41.481896822 (487h22m40.033367283s) [upload=429160]: Starting upload Nov 29 15:26:41.481955112 (487h22m40.033425563s) [upload=429160]: Done starting upload Nov 29 15:26:41.482182997 (487h22m40.033653478s) [upload=429160]: Uploading segments... Nov 29 15:26:41.482310479 (487h22m40.03378094s) [upload=429160]: Got next segment. Inline: false Nov 29 15:26:41.482371305 (487h22m40.033841766s) [upload=429160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:26:41.482377216 (487h22m40.033847677s) [upload=429160]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:26:41.729169431 (487h22m40.280639912s) [seg_pos={0 0}, upload=429160]: Begin upload segment... Nov 29 15:26:41.729234905 (487h22m40.280705376s) [seg_pos={0 0}, upload=429160]: Done begin upload segment. Nov 29 15:26:41.729266475 (487h22m40.280736936s) [noise=true , address=145.79.227.16:4910 , piece_id=VBNI77MAZBTCGFGA37JQUZYQAHL7DCOSBESGPUO5XLJYQQRYYQYA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.729275624 (487h22m40.280746085s) [noise=true , address=45.91.116.9:4262 , piece_id=DLO4CTRY2ZLVM6SLOQ6STR6YC4XW2544EY33BEPGRJSTZUF63THQ, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.729794772 (487h22m40.281265233s) [noise=true , address=145.79.227.21:4475 , piece_id=TSCDHYLYYIR2XHHINAVNT4HF4RQGQV2P6PVCAMXMLIDJMBZ2VQ3Q, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.730079715 (487h22m40.281550176s) [noise=true , address=145.79.227.38:4977 , piece_id=WX2SKHCZOLXK7Y4KJ66U52UDI3GJ2FXU2MBH4WZOHGU5KUARNGUA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.730258976 (487h22m40.281729437s) [noise=true , address=145.79.227.27:4691 , piece_id=NIGF67DRPFG753TH3FQZOSIM3N45KO3B4PDZCNMIOVAYSIZQWKRA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.730408450 (487h22m40.281878911s) [noise=true , address=145.79.227.8:4329 , piece_id=KOZKMMNAYYIPE7FBQRC7H3576SWFW3MZJ5PWCC22WTNPYZOSDFUA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.730652805 (487h22m40.282123256s) [noise=true , address=145.79.227.10:4401 , piece_id=RBUUL3ZYHRRJHTWCGAHBKPOSC2AOZBRAVFYUX46524UDI3BKADHQ, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.730688072 (487h22m40.282158524s) [noise=true , address=145.79.227.44:4194 , piece_id=2FBWCEC2UVGXHMLZJXZBQSPF6NVYOYDBDLLMDFTTBVDE3KRJUALQ, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.730967104 (487h22m40.282437565s) [noise=true , address=145.79.227.20:4439 , piece_id=DSO7HWXUPLYLK5F4YLS7VS5T3U625UJCRRCLR6FLCBY3NHDBQ76A, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.731079858 (487h22m40.282550319s) [noise=true , address=145.79.227.41:4085 , piece_id=AQ2A7OH6LGHOFSBTKXBSNYJLJIPD64DKUBM4KLMXEKFVPHADY7NA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.731240274 (487h22m40.282710725s) [noise=true , address=145.79.227.23:4546 , piece_id=COJF3LRRCB5LEQQLU2U5JSLEKABOJ5AY2T2OTFOIWLYRLXHCW6IA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.731504167 (487h22m40.282974628s) [noise=true , address=145.79.227.4:4185 , piece_id=OFSBQCOFCYXCCROBNWGLZG4VO5HMRCDUM5PXMAQWOKRWHUVP6FJQ, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.731521680 (487h22m40.282992131s) [noise=true , address=145.79.227.19:4079 , piece_id=3Q2LHIVJOHVK5WYGDN5GOO77XVAESHSIUDRGA2XNVA3OVV4V6ASQ, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.731979351 (487h22m40.283449812s) [noise=true , address=145.79.227.15:4876 , piece_id=3E3KI336752SN6U4WS3EXSRVZ4I2HSMXIPPYTX2OS4CTP66TWAWQ, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.732070886 (487h22m40.283541347s) [noise=true , address=45.91.116.5:4299 , piece_id=SXOKIMBIXVYQTTBJV4PMXEHLLSQEBWQGEPU6JLHYRJKZZSEHZZUA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:41.732458414 (487h22m40.283928905s) [noise=true , address=145.79.227.43:4159 , piece_id=XZUVACHL2QYJS2RUSH5SFLOBFCJL27BGL5BQZV5YNQHN4AAKTRGA, seg_pos={0 0}, upload=429160]: Uploading piece... Nov 29 15:26:43.532207571 (487h22m42.083678032s) [upload=429160]: Next returned nil segment Nov 29 15:26:43.532226867 (487h22m42.083697328s) [upload=429160]: Waiting for error group managing segments... Nov 29 15:26:43.836379197 (487h22m42.387849648s) [noise=true , address=45.91.116.5:4299 , piece_id=SXOKIMBIXVYQTTBJV4PMXEHLLSQEBWQGEPU6JLHYRJKZZSEHZZUA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:43.874486060 (487h22m42.425956521s) [noise=true , address=45.91.116.9:4262 , piece_id=DLO4CTRY2ZLVM6SLOQ6STR6YC4XW2544EY33BEPGRJSTZUF63THQ, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:45.758097878 (487h22m44.309568339s) [noise=true , address=145.79.227.19:4079 , piece_id=3Q2LHIVJOHVK5WYGDN5GOO77XVAESHSIUDRGA2XNVA3OVV4V6ASQ, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:45.759427410 (487h22m44.310897871s) [noise=true , address=145.79.227.20:4439 , piece_id=DSO7HWXUPLYLK5F4YLS7VS5T3U625UJCRRCLR6FLCBY3NHDBQ76A, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:45.812204188 (487h22m44.363674649s) [noise=true , address=145.79.227.27:4691 , piece_id=NIGF67DRPFG753TH3FQZOSIM3N45KO3B4PDZCNMIOVAYSIZQWKRA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:45.824856100 (487h22m44.376326561s) [noise=true , address=145.79.227.21:4475 , piece_id=TSCDHYLYYIR2XHHINAVNT4HF4RQGQV2P6PVCAMXMLIDJMBZ2VQ3Q, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:45.896769929 (487h22m44.44824039s) [noise=true , address=145.79.227.16:4910 , piece_id=VBNI77MAZBTCGFGA37JQUZYQAHL7DCOSBESGPUO5XLJYQQRYYQYA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:47.596076674 (487h22m46.147547125s) [noise=true , address=145.79.227.23:4546 , piece_id=COJF3LRRCB5LEQQLU2U5JSLEKABOJ5AY2T2OTFOIWLYRLXHCW6IA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:48.569567674 (487h22m47.121038135s) [noise=true , address=145.79.227.43:4159 , piece_id=XZUVACHL2QYJS2RUSH5SFLOBFCJL27BGL5BQZV5YNQHN4AAKTRGA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:48.695021133 (487h22m47.246491594s) [noise=true , address=145.79.227.41:4085 , piece_id=AQ2A7OH6LGHOFSBTKXBSNYJLJIPD64DKUBM4KLMXEKFVPHADY7NA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:49.035503100 (487h22m47.586973551s) [noise=true , address=145.79.227.44:4194 , piece_id=2FBWCEC2UVGXHMLZJXZBQSPF6NVYOYDBDLLMDFTTBVDE3KRJUALQ, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:50.485461443 (487h22m49.036931905s) [noise=true , address=145.79.227.4:4185 , piece_id=OFSBQCOFCYXCCROBNWGLZG4VO5HMRCDUM5PXMAQWOKRWHUVP6FJQ, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:51.117998887 (487h22m49.669469358s) [noise=true , address=145.79.227.15:4876 , piece_id=3E3KI336752SN6U4WS3EXSRVZ4I2HSMXIPPYTX2OS4CTP66TWAWQ, seg_pos={0 0}, upload=429160]: Done uploading piece. err: <nil> Nov 29 15:26:51.118022191 (487h22m49.66953834s) [seg_pos={0 0}, upload=429160]: Segment reached optimal threshold of 13 pieces. Nov 29 15:26:51.118374071 (487h22m49.669844532s) [noise=true , address=145.79.227.10:4401 , piece_id=RBUUL3ZYHRRJHTWCGAHBKPOSC2AOZBRAVFYUX46524UDI3BKADHQ, seg_pos={0 0}, upload=429160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Nov 29 15:26:51.118385964 (487h22m49.669856425s) [noise=true , address=145.79.227.10:4401 , piece_id=RBUUL3ZYHRRJHTWCGAHBKPOSC2AOZBRAVFYUX46524UDI3BKADHQ, seg_pos={0 0}, upload=429160]: Long tail cancellation due to optimal threshold Nov 29 15:26:51.118573571 (487h22m49.670044032s) [noise=true , address=145.79.227.8:4329 , piece_id=KOZKMMNAYYIPE7FBQRC7H3576SWFW3MZJ5PWCC22WTNPYZOSDFUA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Nov 29 15:26:51.118595002 (487h22m49.670065463s) [noise=true , address=145.79.227.8:4329 , piece_id=KOZKMMNAYYIPE7FBQRC7H3576SWFW3MZJ5PWCC22WTNPYZOSDFUA, seg_pos={0 0}, upload=429160]: Long tail cancellation due to optimal threshold Nov 29 15:26:51.118895474 (487h22m49.670365945s) [noise=true , address=145.79.227.38:4977 , piece_id=WX2SKHCZOLXK7Y4KJ66U52UDI3GJ2FXU2MBH4WZOHGU5KUARNGUA, seg_pos={0 0}, upload=429160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Nov 29 15:26:51.118906756 (487h22m49.670377217s) [noise=true , address=145.79.227.38:4977 , piece_id=WX2SKHCZOLXK7Y4KJ66U52UDI3GJ2FXU2MBH4WZOHGU5KUARNGUA, seg_pos={0 0}, upload=429160]: Long tail cancellation due to optimal threshold Nov 29 15:26:51.118941863 (487h22m49.670412324s) [seg_pos={0 0}, upload=429160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:26:51.119051942 (487h22m49.670522403s) [upload=429160]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:26:51.119076790 (487h22m49.670547251s) [upload=429160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:26:51.833210599 (487h22m50.38468105s) [upload=429160]: Done uploading segments... Nov 29 15:26:51.833226039 (487h22m50.38469649s) [upload=429160]: Upload finished. err: <nil> Duration: 30.004741461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.......X......] duration: 10.495927353s trace: https://jaeger.dev.storj.io/trace/3c12e61589dba0b7 FAIL (2025-11-29 15:26:56 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 34) Duration: 10.495970846s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X........X........................X] duration: 5.001330736s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:26:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001369852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.......X.....] duration: 5.933644146s trace: https://jaeger.dev.storj.io/trace/3c12e61589dba0b7 FAIL (2025-11-29 15:26:08 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 34) Duration: 5.933680075s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.765809092s trace: https://jaeger.dev.storj.io/trace/6a0512aac999f7c2 FAIL (2025-11-29 15:25:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Nov 29 15:22:51.105612271 (487h18m49.657082722s) [upload=429106]: Starting upload Nov 29 15:22:51.105745504 (487h18m49.657215966s) [upload=429106]: Done starting upload Nov 29 15:22:51.105869530 (487h18m49.657340001s) [upload=429106]: Uploading segments... Nov 29 15:22:51.106085461 (487h18m49.657555932s) [upload=429106]: Got next segment. Inline: false Nov 29 15:22:51.106131910 (487h18m49.657602371s) [upload=429106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:22:51.106138713 (487h18m49.657609174s) [upload=429106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:23:09.590060779 (487h19m8.14153125s) [seg_pos={0 0}, upload=429106]: Begin upload segment... Nov 29 15:23:09.590157954 (487h19m8.141628415s) [seg_pos={0 0}, upload=429106]: Done begin upload segment. Nov 29 15:23:09.590233237 (487h19m8.141703688s) [noise=true , address=145.79.227.29:4943 , piece_id=O2O42VAHN447TZGTZBAIEMK4XHL2HAJLUAYLCDYUS22DOU5JXZVA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.590245551 (487h19m8.141716013s) [noise=true , address=45.91.116.13:4153 , piece_id=ZT5WYCSN7IEHSK7IDFDH45KIDKCNCAIKLGZTLDH4DWKVD6X4NTOA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.590887443 (487h19m8.142357934s) [noise=true , address=145.79.227.24:4583 , piece_id=CFBVKC4JG3QHBY7TCAGS3LLP3VUXV5ZXB5KICXVWV6QOGDNXRRVA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.591183357 (487h19m8.142653818s) [noise=true , address=145.79.227.30:4761 , piece_id=7EYTLKY4IWW7OVCRGCLBTOPUVU4P5GBELSPOOWZSBG5OVY6VBKJA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.591626220 (487h19m8.143096681s) [noise=true , address=145.79.227.41:4086 , piece_id=E5TMYSQBHD5VMSFMM7A265QDUFPGCXITPH6YDUQWGDMO4NSG4QJQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.592263424 (487h19m8.143733885s) [noise=true , address=145.79.227.32:4836 , piece_id=VKAXHVZ4FTLN637DD67GHKOQO2JMYUQV6KXEFEFCWMDQP32SMNJA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.592711917 (487h19m8.144182388s) [noise=true , address=145.79.227.44:4195 , piece_id=Q52NMYOH2XMTHW5IYXTEDFZKMYQTLGMR2UPQSLEPVSB4JDTFDQCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.594251249 (487h19m8.14572172s) [noise=true , address=145.79.227.16:4912 , piece_id=RDNJVEPTP7QZDVWVSEOEDXO2S5AAZYMQPMCRYPKO6GGWJ75AJPNQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.595387392 (487h19m8.146857853s) [noise=true , address=145.79.227.21:4474 , piece_id=EBVO3MWK2F75O55XY3OTESSKRVRVHIQUXDYDZLQDEEEIN5XWI23A, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.596616773 (487h19m8.148087234s) [noise=true , address=145.79.227.2:4114 , piece_id=URFVLKJBNT6KVCUWMRMYG4FRD3FPDLBXJIYOH7U4N3DRKJV34GCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.597228698 (487h19m8.148699159s) [noise=true , address=145.79.227.23:4547 , piece_id=6HYKJLNDMFJ3VALCPLNU2XMCJNLGUMH54RTB3YWIE2IXFOXVVXEA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.598031937 (487h19m8.149502409s) [noise=true , address=77.90.31.12:4589 , piece_id=UMQZWOVFLEPLIXYF67FRUGBO5QBL6WNWBUSQOOYKEFOTMJI6KTTQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.598875835 (487h19m8.150346296s) [noise=true , address=45.91.116.14:4191 , piece_id=BSNGKTEMAEGTRMXPJLUH6WTYDXSRXFHVPY5TTA4RDY64TUI6UCWA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.599785808 (487h19m8.151256269s) [noise=true , address=145.79.227.27:4690 , piece_id=5LV442DU3LJ43WBYPWZNY4IQR7KLYRTGMKZUHZ4BCFNMGBA5HO4Q, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.600866045 (487h19m8.152336506s) [noise=true , address=145.79.227.34:4907 , piece_id=DKYEEY3SYJ6W2EZ254VSTULYI22KATGLPWAWJDEHXQ7JWWRBC5LQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.602684687 (487h19m8.154155148s) [noise=true , address=145.79.227.15:4874 , piece_id=QP372DLTT7OPOHA4WG4LVLAGDDMZRD4LVI5LYPWNNWUZ5IKVYYCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:11.338786467 (487h19m9.890256918s) [upload=429106]: Next returned nil segment Nov 29 15:23:11.338800453 (487h19m9.890270904s) [upload=429106]: Waiting for error group managing segments... Nov 29 15:23:11.681354308 (487h19m10.232824759s) [noise=true , address=77.90.31.12:4589 , piece_id=UMQZWOVFLEPLIXYF67FRUGBO5QBL6WNWBUSQOOYKEFOTMJI6KTTQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:11.703681523 (487h19m10.255151984s) [noise=true , address=45.91.116.14:4191 , piece_id=BSNGKTEMAEGTRMXPJLUH6WTYDXSRXFHVPY5TTA4RDY64TUI6UCWA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:11.779054339 (487h19m10.3305248s) [noise=true , address=45.91.116.13:4153 , piece_id=ZT5WYCSN7IEHSK7IDFDH45KIDKCNCAIKLGZTLDH4DWKVD6X4NTOA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.716609475 (487h19m14.268079936s) [noise=true , address=145.79.227.27:4690 , piece_id=5LV442DU3LJ43WBYPWZNY4IQR7KLYRTGMKZUHZ4BCFNMGBA5HO4Q, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.760890015 (487h19m14.312360476s) [noise=true , address=145.79.227.16:4912 , piece_id=RDNJVEPTP7QZDVWVSEOEDXO2S5AAZYMQPMCRYPKO6GGWJ75AJPNQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.776354972 (487h19m14.327825444s) [noise=true , address=145.79.227.30:4761 , piece_id=7EYTLKY4IWW7OVCRGCLBTOPUVU4P5GBELSPOOWZSBG5OVY6VBKJA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:16.578112737 (487h19m15.129583198s) [noise=true , address=145.79.227.29:4943 , piece_id=O2O42VAHN447TZGTZBAIEMK4XHL2HAJLUAYLCDYUS22DOU5JXZVA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:16.795724124 (487h19m15.347194595s) [noise=true , address=145.79.227.15:4874 , piece_id=QP372DLTT7OPOHA4WG4LVLAGDDMZRD4LVI5LYPWNNWUZ5IKVYYCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:17.188868092 (487h19m15.740338553s) [noise=true , address=145.79.227.2:4114 , piece_id=URFVLKJBNT6KVCUWMRMYG4FRD3FPDLBXJIYOH7U4N3DRKJV34GCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:17.422629773 (487h19m15.974100234s) [noise=true , address=145.79.227.44:4195 , piece_id=Q52NMYOH2XMTHW5IYXTEDFZKMYQTLGMR2UPQSLEPVSB4JDTFDQCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:19.050499527 (487h19m17.601969998s) [noise=true , address=145.79.227.23:4547 , piece_id=6HYKJLNDMFJ3VALCPLNU2XMCJNLGUMH54RTB3YWIE2IXFOXVVXEA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050547709 (487h19m17.60201817s) [noise=true , address=145.79.227.34:4907 , piece_id=DKYEEY3SYJ6W2EZ254VSTULYI22KATGLPWAWJDEHXQ7JWWRBC5LQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050804899 (487h19m17.60227537s) [noise=true , address=145.79.227.41:4086 , piece_id=E5TMYSQBHD5VMSFMM7A265QDUFPGCXITPH6YDUQWGDMO4NSG4QJQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050869332 (487h19m17.602339793s) [noise=true , address=145.79.227.21:4474 , piece_id=EBVO3MWK2F75O55XY3OTESSKRVRVHIQUXDYDZLQDEEEIN5XWI23A, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection Nov 29 15:23:19.051003026 (487h19m17.602473497s) [noise=true , address=145.79.227.32:4836 , piece_id=VKAXHVZ4FTLN637DD67GHKOQO2JMYUQV6KXEFEFCWMDQP32SMNJA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.051332694 (487h19m17.602803155s) [noise=true , address=145.79.227.24:4583 , piece_id=CFBVKC4JG3QHBY7TCAGS3LLP3VUXV5ZXB5KICXVWV6QOGDNXRRVA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:49260->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:23:19.051374503 (487h19m17.602844974s) [seg_pos={0 0}, upload=429106]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 15:23:47.596917488 (487h19m46.148387949s) [upload=429106]: Done uploading segments... Nov 29 15:23:47.596962294 (487h19m46.148432755s) [upload=429106]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.765838198s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.......X....] duration: 5.88321031s trace: https://jaeger.dev.storj.io/trace/3c12e61589dba0b7 FAIL (2025-11-29 15:25:49 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 34) Duration: 5.883236s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.976833453s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:25:38 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.976865845s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X........X........................] duration: 3.471822263s trace: https://jaeger.dev.storj.io/trace/66b8eb4790a4cf18 FAIL (2025-11-29 15:25:37 UTC) orbiter warning: context deadline exceeded Duration: 3.47188399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.77386219s trace: https://jaeger.dev.storj.io/trace/6a0512aac999f7c2 FAIL (2025-11-29 15:25:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Nov 29 15:22:51.105612271 (487h18m49.657082722s) [upload=429106]: Starting upload Nov 29 15:22:51.105745504 (487h18m49.657215966s) [upload=429106]: Done starting upload Nov 29 15:22:51.105869530 (487h18m49.657340001s) [upload=429106]: Uploading segments... Nov 29 15:22:51.106085461 (487h18m49.657555932s) [upload=429106]: Got next segment. Inline: false Nov 29 15:22:51.106131910 (487h18m49.657602371s) [upload=429106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:22:51.106138713 (487h18m49.657609174s) [upload=429106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:23:09.590060779 (487h19m8.14153125s) [seg_pos={0 0}, upload=429106]: Begin upload segment... Nov 29 15:23:09.590157954 (487h19m8.141628415s) [seg_pos={0 0}, upload=429106]: Done begin upload segment. Nov 29 15:23:09.590233237 (487h19m8.141703688s) [noise=true , address=145.79.227.29:4943 , piece_id=O2O42VAHN447TZGTZBAIEMK4XHL2HAJLUAYLCDYUS22DOU5JXZVA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.590245551 (487h19m8.141716013s) [noise=true , address=45.91.116.13:4153 , piece_id=ZT5WYCSN7IEHSK7IDFDH45KIDKCNCAIKLGZTLDH4DWKVD6X4NTOA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.590887443 (487h19m8.142357934s) [noise=true , address=145.79.227.24:4583 , piece_id=CFBVKC4JG3QHBY7TCAGS3LLP3VUXV5ZXB5KICXVWV6QOGDNXRRVA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.591183357 (487h19m8.142653818s) [noise=true , address=145.79.227.30:4761 , piece_id=7EYTLKY4IWW7OVCRGCLBTOPUVU4P5GBELSPOOWZSBG5OVY6VBKJA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.591626220 (487h19m8.143096681s) [noise=true , address=145.79.227.41:4086 , piece_id=E5TMYSQBHD5VMSFMM7A265QDUFPGCXITPH6YDUQWGDMO4NSG4QJQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.592263424 (487h19m8.143733885s) [noise=true , address=145.79.227.32:4836 , piece_id=VKAXHVZ4FTLN637DD67GHKOQO2JMYUQV6KXEFEFCWMDQP32SMNJA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.592711917 (487h19m8.144182388s) [noise=true , address=145.79.227.44:4195 , piece_id=Q52NMYOH2XMTHW5IYXTEDFZKMYQTLGMR2UPQSLEPVSB4JDTFDQCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.594251249 (487h19m8.14572172s) [noise=true , address=145.79.227.16:4912 , piece_id=RDNJVEPTP7QZDVWVSEOEDXO2S5AAZYMQPMCRYPKO6GGWJ75AJPNQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.595387392 (487h19m8.146857853s) [noise=true , address=145.79.227.21:4474 , piece_id=EBVO3MWK2F75O55XY3OTESSKRVRVHIQUXDYDZLQDEEEIN5XWI23A, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.596616773 (487h19m8.148087234s) [noise=true , address=145.79.227.2:4114 , piece_id=URFVLKJBNT6KVCUWMRMYG4FRD3FPDLBXJIYOH7U4N3DRKJV34GCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.597228698 (487h19m8.148699159s) [noise=true , address=145.79.227.23:4547 , piece_id=6HYKJLNDMFJ3VALCPLNU2XMCJNLGUMH54RTB3YWIE2IXFOXVVXEA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.598031937 (487h19m8.149502409s) [noise=true , address=77.90.31.12:4589 , piece_id=UMQZWOVFLEPLIXYF67FRUGBO5QBL6WNWBUSQOOYKEFOTMJI6KTTQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.598875835 (487h19m8.150346296s) [noise=true , address=45.91.116.14:4191 , piece_id=BSNGKTEMAEGTRMXPJLUH6WTYDXSRXFHVPY5TTA4RDY64TUI6UCWA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.599785808 (487h19m8.151256269s) [noise=true , address=145.79.227.27:4690 , piece_id=5LV442DU3LJ43WBYPWZNY4IQR7KLYRTGMKZUHZ4BCFNMGBA5HO4Q, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.600866045 (487h19m8.152336506s) [noise=true , address=145.79.227.34:4907 , piece_id=DKYEEY3SYJ6W2EZ254VSTULYI22KATGLPWAWJDEHXQ7JWWRBC5LQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.602684687 (487h19m8.154155148s) [noise=true , address=145.79.227.15:4874 , piece_id=QP372DLTT7OPOHA4WG4LVLAGDDMZRD4LVI5LYPWNNWUZ5IKVYYCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:11.338786467 (487h19m9.890256918s) [upload=429106]: Next returned nil segment Nov 29 15:23:11.338800453 (487h19m9.890270904s) [upload=429106]: Waiting for error group managing segments... Nov 29 15:23:11.681354308 (487h19m10.232824759s) [noise=true , address=77.90.31.12:4589 , piece_id=UMQZWOVFLEPLIXYF67FRUGBO5QBL6WNWBUSQOOYKEFOTMJI6KTTQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:11.703681523 (487h19m10.255151984s) [noise=true , address=45.91.116.14:4191 , piece_id=BSNGKTEMAEGTRMXPJLUH6WTYDXSRXFHVPY5TTA4RDY64TUI6UCWA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:11.779054339 (487h19m10.3305248s) [noise=true , address=45.91.116.13:4153 , piece_id=ZT5WYCSN7IEHSK7IDFDH45KIDKCNCAIKLGZTLDH4DWKVD6X4NTOA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.716609475 (487h19m14.268079936s) [noise=true , address=145.79.227.27:4690 , piece_id=5LV442DU3LJ43WBYPWZNY4IQR7KLYRTGMKZUHZ4BCFNMGBA5HO4Q, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.760890015 (487h19m14.312360476s) [noise=true , address=145.79.227.16:4912 , piece_id=RDNJVEPTP7QZDVWVSEOEDXO2S5AAZYMQPMCRYPKO6GGWJ75AJPNQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.776354972 (487h19m14.327825444s) [noise=true , address=145.79.227.30:4761 , piece_id=7EYTLKY4IWW7OVCRGCLBTOPUVU4P5GBELSPOOWZSBG5OVY6VBKJA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:16.578112737 (487h19m15.129583198s) [noise=true , address=145.79.227.29:4943 , piece_id=O2O42VAHN447TZGTZBAIEMK4XHL2HAJLUAYLCDYUS22DOU5JXZVA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:16.795724124 (487h19m15.347194595s) [noise=true , address=145.79.227.15:4874 , piece_id=QP372DLTT7OPOHA4WG4LVLAGDDMZRD4LVI5LYPWNNWUZ5IKVYYCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:17.188868092 (487h19m15.740338553s) [noise=true , address=145.79.227.2:4114 , piece_id=URFVLKJBNT6KVCUWMRMYG4FRD3FPDLBXJIYOH7U4N3DRKJV34GCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:17.422629773 (487h19m15.974100234s) [noise=true , address=145.79.227.44:4195 , piece_id=Q52NMYOH2XMTHW5IYXTEDFZKMYQTLGMR2UPQSLEPVSB4JDTFDQCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:19.050499527 (487h19m17.601969998s) [noise=true , address=145.79.227.23:4547 , piece_id=6HYKJLNDMFJ3VALCPLNU2XMCJNLGUMH54RTB3YWIE2IXFOXVVXEA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050547709 (487h19m17.60201817s) [noise=true , address=145.79.227.34:4907 , piece_id=DKYEEY3SYJ6W2EZ254VSTULYI22KATGLPWAWJDEHXQ7JWWRBC5LQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050804899 (487h19m17.60227537s) [noise=true , address=145.79.227.41:4086 , piece_id=E5TMYSQBHD5VMSFMM7A265QDUFPGCXITPH6YDUQWGDMO4NSG4QJQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050869332 (487h19m17.602339793s) [noise=true , address=145.79.227.21:4474 , piece_id=EBVO3MWK2F75O55XY3OTESSKRVRVHIQUXDYDZLQDEEEIN5XWI23A, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection Nov 29 15:23:19.051003026 (487h19m17.602473497s) [noise=true , address=145.79.227.32:4836 , piece_id=VKAXHVZ4FTLN637DD67GHKOQO2JMYUQV6KXEFEFCWMDQP32SMNJA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.051332694 (487h19m17.602803155s) [noise=true , address=145.79.227.24:4583 , piece_id=CFBVKC4JG3QHBY7TCAGS3LLP3VUXV5ZXB5KICXVWV6QOGDNXRRVA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:49260->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:23:19.051374503 (487h19m17.602844974s) [seg_pos={0 0}, upload=429106]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 15:23:47.596917488 (487h19m46.148387949s) [upload=429106]: Done uploading segments... Nov 29 15:23:47.596962294 (487h19m46.148432755s) [upload=429106]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.773898489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.......X...] duration: 6.342505163s trace: https://jaeger.dev.storj.io/trace/3c12e61589dba0b7 FAIL (2025-11-29 15:24:59 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 34) Duration: 6.342533429s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X........X.......................] duration: 3.428222292s trace: https://jaeger.dev.storj.io/trace/66b8eb4790a4cf18 FAIL (2025-11-29 15:24:49 UTC) orbiter warning: context deadline exceeded Duration: 3.428252259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.......X..] duration: 6.061290059s trace: https://jaeger.dev.storj.io/trace/3c12e61589dba0b7 FAIL (2025-11-29 15:24:44 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 34) Duration: 6.061350263s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.956538556s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:24:36 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.956571339s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.818729586s trace: https://jaeger.dev.storj.io/trace/6a0512aac999f7c2 FAIL (2025-11-29 15:24:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Nov 29 15:22:51.105612271 (487h18m49.657082722s) [upload=429106]: Starting upload Nov 29 15:22:51.105745504 (487h18m49.657215966s) [upload=429106]: Done starting upload Nov 29 15:22:51.105869530 (487h18m49.657340001s) [upload=429106]: Uploading segments... Nov 29 15:22:51.106085461 (487h18m49.657555932s) [upload=429106]: Got next segment. Inline: false Nov 29 15:22:51.106131910 (487h18m49.657602371s) [upload=429106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:22:51.106138713 (487h18m49.657609174s) [upload=429106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:23:09.590060779 (487h19m8.14153125s) [seg_pos={0 0}, upload=429106]: Begin upload segment... Nov 29 15:23:09.590157954 (487h19m8.141628415s) [seg_pos={0 0}, upload=429106]: Done begin upload segment. Nov 29 15:23:09.590233237 (487h19m8.141703688s) [noise=true , address=145.79.227.29:4943 , piece_id=O2O42VAHN447TZGTZBAIEMK4XHL2HAJLUAYLCDYUS22DOU5JXZVA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.590245551 (487h19m8.141716013s) [noise=true , address=45.91.116.13:4153 , piece_id=ZT5WYCSN7IEHSK7IDFDH45KIDKCNCAIKLGZTLDH4DWKVD6X4NTOA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.590887443 (487h19m8.142357934s) [noise=true , address=145.79.227.24:4583 , piece_id=CFBVKC4JG3QHBY7TCAGS3LLP3VUXV5ZXB5KICXVWV6QOGDNXRRVA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.591183357 (487h19m8.142653818s) [noise=true , address=145.79.227.30:4761 , piece_id=7EYTLKY4IWW7OVCRGCLBTOPUVU4P5GBELSPOOWZSBG5OVY6VBKJA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.591626220 (487h19m8.143096681s) [noise=true , address=145.79.227.41:4086 , piece_id=E5TMYSQBHD5VMSFMM7A265QDUFPGCXITPH6YDUQWGDMO4NSG4QJQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.592263424 (487h19m8.143733885s) [noise=true , address=145.79.227.32:4836 , piece_id=VKAXHVZ4FTLN637DD67GHKOQO2JMYUQV6KXEFEFCWMDQP32SMNJA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.592711917 (487h19m8.144182388s) [noise=true , address=145.79.227.44:4195 , piece_id=Q52NMYOH2XMTHW5IYXTEDFZKMYQTLGMR2UPQSLEPVSB4JDTFDQCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.594251249 (487h19m8.14572172s) [noise=true , address=145.79.227.16:4912 , piece_id=RDNJVEPTP7QZDVWVSEOEDXO2S5AAZYMQPMCRYPKO6GGWJ75AJPNQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.595387392 (487h19m8.146857853s) [noise=true , address=145.79.227.21:4474 , piece_id=EBVO3MWK2F75O55XY3OTESSKRVRVHIQUXDYDZLQDEEEIN5XWI23A, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.596616773 (487h19m8.148087234s) [noise=true , address=145.79.227.2:4114 , piece_id=URFVLKJBNT6KVCUWMRMYG4FRD3FPDLBXJIYOH7U4N3DRKJV34GCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.597228698 (487h19m8.148699159s) [noise=true , address=145.79.227.23:4547 , piece_id=6HYKJLNDMFJ3VALCPLNU2XMCJNLGUMH54RTB3YWIE2IXFOXVVXEA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.598031937 (487h19m8.149502409s) [noise=true , address=77.90.31.12:4589 , piece_id=UMQZWOVFLEPLIXYF67FRUGBO5QBL6WNWBUSQOOYKEFOTMJI6KTTQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.598875835 (487h19m8.150346296s) [noise=true , address=45.91.116.14:4191 , piece_id=BSNGKTEMAEGTRMXPJLUH6WTYDXSRXFHVPY5TTA4RDY64TUI6UCWA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.599785808 (487h19m8.151256269s) [noise=true , address=145.79.227.27:4690 , piece_id=5LV442DU3LJ43WBYPWZNY4IQR7KLYRTGMKZUHZ4BCFNMGBA5HO4Q, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.600866045 (487h19m8.152336506s) [noise=true , address=145.79.227.34:4907 , piece_id=DKYEEY3SYJ6W2EZ254VSTULYI22KATGLPWAWJDEHXQ7JWWRBC5LQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.602684687 (487h19m8.154155148s) [noise=true , address=145.79.227.15:4874 , piece_id=QP372DLTT7OPOHA4WG4LVLAGDDMZRD4LVI5LYPWNNWUZ5IKVYYCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:11.338786467 (487h19m9.890256918s) [upload=429106]: Next returned nil segment Nov 29 15:23:11.338800453 (487h19m9.890270904s) [upload=429106]: Waiting for error group managing segments... Nov 29 15:23:11.681354308 (487h19m10.232824759s) [noise=true , address=77.90.31.12:4589 , piece_id=UMQZWOVFLEPLIXYF67FRUGBO5QBL6WNWBUSQOOYKEFOTMJI6KTTQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:11.703681523 (487h19m10.255151984s) [noise=true , address=45.91.116.14:4191 , piece_id=BSNGKTEMAEGTRMXPJLUH6WTYDXSRXFHVPY5TTA4RDY64TUI6UCWA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:11.779054339 (487h19m10.3305248s) [noise=true , address=45.91.116.13:4153 , piece_id=ZT5WYCSN7IEHSK7IDFDH45KIDKCNCAIKLGZTLDH4DWKVD6X4NTOA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.716609475 (487h19m14.268079936s) [noise=true , address=145.79.227.27:4690 , piece_id=5LV442DU3LJ43WBYPWZNY4IQR7KLYRTGMKZUHZ4BCFNMGBA5HO4Q, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.760890015 (487h19m14.312360476s) [noise=true , address=145.79.227.16:4912 , piece_id=RDNJVEPTP7QZDVWVSEOEDXO2S5AAZYMQPMCRYPKO6GGWJ75AJPNQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.776354972 (487h19m14.327825444s) [noise=true , address=145.79.227.30:4761 , piece_id=7EYTLKY4IWW7OVCRGCLBTOPUVU4P5GBELSPOOWZSBG5OVY6VBKJA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:16.578112737 (487h19m15.129583198s) [noise=true , address=145.79.227.29:4943 , piece_id=O2O42VAHN447TZGTZBAIEMK4XHL2HAJLUAYLCDYUS22DOU5JXZVA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:16.795724124 (487h19m15.347194595s) [noise=true , address=145.79.227.15:4874 , piece_id=QP372DLTT7OPOHA4WG4LVLAGDDMZRD4LVI5LYPWNNWUZ5IKVYYCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:17.188868092 (487h19m15.740338553s) [noise=true , address=145.79.227.2:4114 , piece_id=URFVLKJBNT6KVCUWMRMYG4FRD3FPDLBXJIYOH7U4N3DRKJV34GCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:17.422629773 (487h19m15.974100234s) [noise=true , address=145.79.227.44:4195 , piece_id=Q52NMYOH2XMTHW5IYXTEDFZKMYQTLGMR2UPQSLEPVSB4JDTFDQCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:19.050499527 (487h19m17.601969998s) [noise=true , address=145.79.227.23:4547 , piece_id=6HYKJLNDMFJ3VALCPLNU2XMCJNLGUMH54RTB3YWIE2IXFOXVVXEA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050547709 (487h19m17.60201817s) [noise=true , address=145.79.227.34:4907 , piece_id=DKYEEY3SYJ6W2EZ254VSTULYI22KATGLPWAWJDEHXQ7JWWRBC5LQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050804899 (487h19m17.60227537s) [noise=true , address=145.79.227.41:4086 , piece_id=E5TMYSQBHD5VMSFMM7A265QDUFPGCXITPH6YDUQWGDMO4NSG4QJQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050869332 (487h19m17.602339793s) [noise=true , address=145.79.227.21:4474 , piece_id=EBVO3MWK2F75O55XY3OTESSKRVRVHIQUXDYDZLQDEEEIN5XWI23A, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection Nov 29 15:23:19.051003026 (487h19m17.602473497s) [noise=true , address=145.79.227.32:4836 , piece_id=VKAXHVZ4FTLN637DD67GHKOQO2JMYUQV6KXEFEFCWMDQP32SMNJA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.051332694 (487h19m17.602803155s) [noise=true , address=145.79.227.24:4583 , piece_id=CFBVKC4JG3QHBY7TCAGS3LLP3VUXV5ZXB5KICXVWV6QOGDNXRRVA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:49260->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:23:19.051374503 (487h19m17.602844974s) [seg_pos={0 0}, upload=429106]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 15:23:47.596917488 (487h19m46.148387949s) [upload=429106]: Done uploading segments... Nov 29 15:23:47.596962294 (487h19m46.148432755s) [upload=429106]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.818759722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.......X.] duration: 6.17960086s trace: https://jaeger.dev.storj.io/trace/3c12e61589dba0b7 FAIL (2025-11-29 15:24:12 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 34) Duration: 6.179636139s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X........X......................] duration: 5.002424236s trace: https://jaeger.dev.storj.io/trace/66b8eb4790a4cf18 FAIL (2025-11-29 15:24:03 UTC) orbiter warning: context deadline exceeded Duration: 5.002451728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.828680252s trace: https://jaeger.dev.storj.io/trace/6a0512aac999f7c2 FAIL (2025-11-29 15:23:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Nov 29 15:22:51.105612271 (487h18m49.657082722s) [upload=429106]: Starting upload Nov 29 15:22:51.105745504 (487h18m49.657215966s) [upload=429106]: Done starting upload Nov 29 15:22:51.105869530 (487h18m49.657340001s) [upload=429106]: Uploading segments... Nov 29 15:22:51.106085461 (487h18m49.657555932s) [upload=429106]: Got next segment. Inline: false Nov 29 15:22:51.106131910 (487h18m49.657602371s) [upload=429106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:22:51.106138713 (487h18m49.657609174s) [upload=429106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:23:09.590060779 (487h19m8.14153125s) [seg_pos={0 0}, upload=429106]: Begin upload segment... Nov 29 15:23:09.590157954 (487h19m8.141628415s) [seg_pos={0 0}, upload=429106]: Done begin upload segment. Nov 29 15:23:09.590233237 (487h19m8.141703688s) [noise=true , address=145.79.227.29:4943 , piece_id=O2O42VAHN447TZGTZBAIEMK4XHL2HAJLUAYLCDYUS22DOU5JXZVA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.590245551 (487h19m8.141716013s) [noise=true , address=45.91.116.13:4153 , piece_id=ZT5WYCSN7IEHSK7IDFDH45KIDKCNCAIKLGZTLDH4DWKVD6X4NTOA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.590887443 (487h19m8.142357934s) [noise=true , address=145.79.227.24:4583 , piece_id=CFBVKC4JG3QHBY7TCAGS3LLP3VUXV5ZXB5KICXVWV6QOGDNXRRVA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.591183357 (487h19m8.142653818s) [noise=true , address=145.79.227.30:4761 , piece_id=7EYTLKY4IWW7OVCRGCLBTOPUVU4P5GBELSPOOWZSBG5OVY6VBKJA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.591626220 (487h19m8.143096681s) [noise=true , address=145.79.227.41:4086 , piece_id=E5TMYSQBHD5VMSFMM7A265QDUFPGCXITPH6YDUQWGDMO4NSG4QJQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.592263424 (487h19m8.143733885s) [noise=true , address=145.79.227.32:4836 , piece_id=VKAXHVZ4FTLN637DD67GHKOQO2JMYUQV6KXEFEFCWMDQP32SMNJA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.592711917 (487h19m8.144182388s) [noise=true , address=145.79.227.44:4195 , piece_id=Q52NMYOH2XMTHW5IYXTEDFZKMYQTLGMR2UPQSLEPVSB4JDTFDQCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.594251249 (487h19m8.14572172s) [noise=true , address=145.79.227.16:4912 , piece_id=RDNJVEPTP7QZDVWVSEOEDXO2S5AAZYMQPMCRYPKO6GGWJ75AJPNQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.595387392 (487h19m8.146857853s) [noise=true , address=145.79.227.21:4474 , piece_id=EBVO3MWK2F75O55XY3OTESSKRVRVHIQUXDYDZLQDEEEIN5XWI23A, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.596616773 (487h19m8.148087234s) [noise=true , address=145.79.227.2:4114 , piece_id=URFVLKJBNT6KVCUWMRMYG4FRD3FPDLBXJIYOH7U4N3DRKJV34GCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.597228698 (487h19m8.148699159s) [noise=true , address=145.79.227.23:4547 , piece_id=6HYKJLNDMFJ3VALCPLNU2XMCJNLGUMH54RTB3YWIE2IXFOXVVXEA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.598031937 (487h19m8.149502409s) [noise=true , address=77.90.31.12:4589 , piece_id=UMQZWOVFLEPLIXYF67FRUGBO5QBL6WNWBUSQOOYKEFOTMJI6KTTQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.598875835 (487h19m8.150346296s) [noise=true , address=45.91.116.14:4191 , piece_id=BSNGKTEMAEGTRMXPJLUH6WTYDXSRXFHVPY5TTA4RDY64TUI6UCWA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.599785808 (487h19m8.151256269s) [noise=true , address=145.79.227.27:4690 , piece_id=5LV442DU3LJ43WBYPWZNY4IQR7KLYRTGMKZUHZ4BCFNMGBA5HO4Q, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.600866045 (487h19m8.152336506s) [noise=true , address=145.79.227.34:4907 , piece_id=DKYEEY3SYJ6W2EZ254VSTULYI22KATGLPWAWJDEHXQ7JWWRBC5LQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.602684687 (487h19m8.154155148s) [noise=true , address=145.79.227.15:4874 , piece_id=QP372DLTT7OPOHA4WG4LVLAGDDMZRD4LVI5LYPWNNWUZ5IKVYYCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:11.338786467 (487h19m9.890256918s) [upload=429106]: Next returned nil segment Nov 29 15:23:11.338800453 (487h19m9.890270904s) [upload=429106]: Waiting for error group managing segments... Nov 29 15:23:11.681354308 (487h19m10.232824759s) [noise=true , address=77.90.31.12:4589 , piece_id=UMQZWOVFLEPLIXYF67FRUGBO5QBL6WNWBUSQOOYKEFOTMJI6KTTQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:11.703681523 (487h19m10.255151984s) [noise=true , address=45.91.116.14:4191 , piece_id=BSNGKTEMAEGTRMXPJLUH6WTYDXSRXFHVPY5TTA4RDY64TUI6UCWA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:11.779054339 (487h19m10.3305248s) [noise=true , address=45.91.116.13:4153 , piece_id=ZT5WYCSN7IEHSK7IDFDH45KIDKCNCAIKLGZTLDH4DWKVD6X4NTOA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.716609475 (487h19m14.268079936s) [noise=true , address=145.79.227.27:4690 , piece_id=5LV442DU3LJ43WBYPWZNY4IQR7KLYRTGMKZUHZ4BCFNMGBA5HO4Q, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.760890015 (487h19m14.312360476s) [noise=true , address=145.79.227.16:4912 , piece_id=RDNJVEPTP7QZDVWVSEOEDXO2S5AAZYMQPMCRYPKO6GGWJ75AJPNQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.776354972 (487h19m14.327825444s) [noise=true , address=145.79.227.30:4761 , piece_id=7EYTLKY4IWW7OVCRGCLBTOPUVU4P5GBELSPOOWZSBG5OVY6VBKJA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:16.578112737 (487h19m15.129583198s) [noise=true , address=145.79.227.29:4943 , piece_id=O2O42VAHN447TZGTZBAIEMK4XHL2HAJLUAYLCDYUS22DOU5JXZVA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:16.795724124 (487h19m15.347194595s) [noise=true , address=145.79.227.15:4874 , piece_id=QP372DLTT7OPOHA4WG4LVLAGDDMZRD4LVI5LYPWNNWUZ5IKVYYCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:17.188868092 (487h19m15.740338553s) [noise=true , address=145.79.227.2:4114 , piece_id=URFVLKJBNT6KVCUWMRMYG4FRD3FPDLBXJIYOH7U4N3DRKJV34GCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:17.422629773 (487h19m15.974100234s) [noise=true , address=145.79.227.44:4195 , piece_id=Q52NMYOH2XMTHW5IYXTEDFZKMYQTLGMR2UPQSLEPVSB4JDTFDQCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:19.050499527 (487h19m17.601969998s) [noise=true , address=145.79.227.23:4547 , piece_id=6HYKJLNDMFJ3VALCPLNU2XMCJNLGUMH54RTB3YWIE2IXFOXVVXEA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050547709 (487h19m17.60201817s) [noise=true , address=145.79.227.34:4907 , piece_id=DKYEEY3SYJ6W2EZ254VSTULYI22KATGLPWAWJDEHXQ7JWWRBC5LQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050804899 (487h19m17.60227537s) [noise=true , address=145.79.227.41:4086 , piece_id=E5TMYSQBHD5VMSFMM7A265QDUFPGCXITPH6YDUQWGDMO4NSG4QJQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050869332 (487h19m17.602339793s) [noise=true , address=145.79.227.21:4474 , piece_id=EBVO3MWK2F75O55XY3OTESSKRVRVHIQUXDYDZLQDEEEIN5XWI23A, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection Nov 29 15:23:19.051003026 (487h19m17.602473497s) [noise=true , address=145.79.227.32:4836 , piece_id=VKAXHVZ4FTLN637DD67GHKOQO2JMYUQV6KXEFEFCWMDQP32SMNJA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.051332694 (487h19m17.602803155s) [noise=true , address=145.79.227.24:4583 , piece_id=CFBVKC4JG3QHBY7TCAGS3LLP3VUXV5ZXB5KICXVWV6QOGDNXRRVA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:49260->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:23:19.051374503 (487h19m17.602844974s) [seg_pos={0 0}, upload=429106]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 15:23:47.596917488 (487h19m46.148387949s) [upload=429106]: Done uploading segments... Nov 29 15:23:47.596962294 (487h19m46.148432755s) [upload=429106]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.828700161s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 58.549193556s trace: https://jaeger.dev.storj.io/trace/6a0512aac999f7c2 FAIL (2025-11-29 15:23:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Nov 29 15:22:51.105612271 (487h18m49.657082722s) [upload=429106]: Starting upload Nov 29 15:22:51.105745504 (487h18m49.657215966s) [upload=429106]: Done starting upload Nov 29 15:22:51.105869530 (487h18m49.657340001s) [upload=429106]: Uploading segments... Nov 29 15:22:51.106085461 (487h18m49.657555932s) [upload=429106]: Got next segment. Inline: false Nov 29 15:22:51.106131910 (487h18m49.657602371s) [upload=429106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:22:51.106138713 (487h18m49.657609174s) [upload=429106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:23:09.590060779 (487h19m8.14153125s) [seg_pos={0 0}, upload=429106]: Begin upload segment... Nov 29 15:23:09.590157954 (487h19m8.141628415s) [seg_pos={0 0}, upload=429106]: Done begin upload segment. Nov 29 15:23:09.590233237 (487h19m8.141703688s) [noise=true , address=145.79.227.29:4943 , piece_id=O2O42VAHN447TZGTZBAIEMK4XHL2HAJLUAYLCDYUS22DOU5JXZVA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.590245551 (487h19m8.141716013s) [noise=true , address=45.91.116.13:4153 , piece_id=ZT5WYCSN7IEHSK7IDFDH45KIDKCNCAIKLGZTLDH4DWKVD6X4NTOA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.590887443 (487h19m8.142357934s) [noise=true , address=145.79.227.24:4583 , piece_id=CFBVKC4JG3QHBY7TCAGS3LLP3VUXV5ZXB5KICXVWV6QOGDNXRRVA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.591183357 (487h19m8.142653818s) [noise=true , address=145.79.227.30:4761 , piece_id=7EYTLKY4IWW7OVCRGCLBTOPUVU4P5GBELSPOOWZSBG5OVY6VBKJA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.591626220 (487h19m8.143096681s) [noise=true , address=145.79.227.41:4086 , piece_id=E5TMYSQBHD5VMSFMM7A265QDUFPGCXITPH6YDUQWGDMO4NSG4QJQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.592263424 (487h19m8.143733885s) [noise=true , address=145.79.227.32:4836 , piece_id=VKAXHVZ4FTLN637DD67GHKOQO2JMYUQV6KXEFEFCWMDQP32SMNJA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.592711917 (487h19m8.144182388s) [noise=true , address=145.79.227.44:4195 , piece_id=Q52NMYOH2XMTHW5IYXTEDFZKMYQTLGMR2UPQSLEPVSB4JDTFDQCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.594251249 (487h19m8.14572172s) [noise=true , address=145.79.227.16:4912 , piece_id=RDNJVEPTP7QZDVWVSEOEDXO2S5AAZYMQPMCRYPKO6GGWJ75AJPNQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.595387392 (487h19m8.146857853s) [noise=true , address=145.79.227.21:4474 , piece_id=EBVO3MWK2F75O55XY3OTESSKRVRVHIQUXDYDZLQDEEEIN5XWI23A, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.596616773 (487h19m8.148087234s) [noise=true , address=145.79.227.2:4114 , piece_id=URFVLKJBNT6KVCUWMRMYG4FRD3FPDLBXJIYOH7U4N3DRKJV34GCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.597228698 (487h19m8.148699159s) [noise=true , address=145.79.227.23:4547 , piece_id=6HYKJLNDMFJ3VALCPLNU2XMCJNLGUMH54RTB3YWIE2IXFOXVVXEA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.598031937 (487h19m8.149502409s) [noise=true , address=77.90.31.12:4589 , piece_id=UMQZWOVFLEPLIXYF67FRUGBO5QBL6WNWBUSQOOYKEFOTMJI6KTTQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.598875835 (487h19m8.150346296s) [noise=true , address=45.91.116.14:4191 , piece_id=BSNGKTEMAEGTRMXPJLUH6WTYDXSRXFHVPY5TTA4RDY64TUI6UCWA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.599785808 (487h19m8.151256269s) [noise=true , address=145.79.227.27:4690 , piece_id=5LV442DU3LJ43WBYPWZNY4IQR7KLYRTGMKZUHZ4BCFNMGBA5HO4Q, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.600866045 (487h19m8.152336506s) [noise=true , address=145.79.227.34:4907 , piece_id=DKYEEY3SYJ6W2EZ254VSTULYI22KATGLPWAWJDEHXQ7JWWRBC5LQ, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:09.602684687 (487h19m8.154155148s) [noise=true , address=145.79.227.15:4874 , piece_id=QP372DLTT7OPOHA4WG4LVLAGDDMZRD4LVI5LYPWNNWUZ5IKVYYCA, seg_pos={0 0}, upload=429106]: Uploading piece... Nov 29 15:23:11.338786467 (487h19m9.890256918s) [upload=429106]: Next returned nil segment Nov 29 15:23:11.338800453 (487h19m9.890270904s) [upload=429106]: Waiting for error group managing segments... Nov 29 15:23:11.681354308 (487h19m10.232824759s) [noise=true , address=77.90.31.12:4589 , piece_id=UMQZWOVFLEPLIXYF67FRUGBO5QBL6WNWBUSQOOYKEFOTMJI6KTTQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:11.703681523 (487h19m10.255151984s) [noise=true , address=45.91.116.14:4191 , piece_id=BSNGKTEMAEGTRMXPJLUH6WTYDXSRXFHVPY5TTA4RDY64TUI6UCWA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:11.779054339 (487h19m10.3305248s) [noise=true , address=45.91.116.13:4153 , piece_id=ZT5WYCSN7IEHSK7IDFDH45KIDKCNCAIKLGZTLDH4DWKVD6X4NTOA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.716609475 (487h19m14.268079936s) [noise=true , address=145.79.227.27:4690 , piece_id=5LV442DU3LJ43WBYPWZNY4IQR7KLYRTGMKZUHZ4BCFNMGBA5HO4Q, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.760890015 (487h19m14.312360476s) [noise=true , address=145.79.227.16:4912 , piece_id=RDNJVEPTP7QZDVWVSEOEDXO2S5AAZYMQPMCRYPKO6GGWJ75AJPNQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:15.776354972 (487h19m14.327825444s) [noise=true , address=145.79.227.30:4761 , piece_id=7EYTLKY4IWW7OVCRGCLBTOPUVU4P5GBELSPOOWZSBG5OVY6VBKJA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:16.578112737 (487h19m15.129583198s) [noise=true , address=145.79.227.29:4943 , piece_id=O2O42VAHN447TZGTZBAIEMK4XHL2HAJLUAYLCDYUS22DOU5JXZVA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:16.795724124 (487h19m15.347194595s) [noise=true , address=145.79.227.15:4874 , piece_id=QP372DLTT7OPOHA4WG4LVLAGDDMZRD4LVI5LYPWNNWUZ5IKVYYCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:17.188868092 (487h19m15.740338553s) [noise=true , address=145.79.227.2:4114 , piece_id=URFVLKJBNT6KVCUWMRMYG4FRD3FPDLBXJIYOH7U4N3DRKJV34GCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:17.422629773 (487h19m15.974100234s) [noise=true , address=145.79.227.44:4195 , piece_id=Q52NMYOH2XMTHW5IYXTEDFZKMYQTLGMR2UPQSLEPVSB4JDTFDQCA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: <nil> Nov 29 15:23:19.050499527 (487h19m17.601969998s) [noise=true , address=145.79.227.23:4547 , piece_id=6HYKJLNDMFJ3VALCPLNU2XMCJNLGUMH54RTB3YWIE2IXFOXVVXEA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050547709 (487h19m17.60201817s) [noise=true , address=145.79.227.34:4907 , piece_id=DKYEEY3SYJ6W2EZ254VSTULYI22KATGLPWAWJDEHXQ7JWWRBC5LQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050804899 (487h19m17.60227537s) [noise=true , address=145.79.227.41:4086 , piece_id=E5TMYSQBHD5VMSFMM7A265QDUFPGCXITPH6YDUQWGDMO4NSG4QJQ, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.050869332 (487h19m17.602339793s) [noise=true , address=145.79.227.21:4474 , piece_id=EBVO3MWK2F75O55XY3OTESSKRVRVHIQUXDYDZLQDEEEIN5XWI23A, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49422->145.79.227.21:4474: use of closed network connection Nov 29 15:23:19.051003026 (487h19m17.602473497s) [noise=true , address=145.79.227.32:4836 , piece_id=VKAXHVZ4FTLN637DD67GHKOQO2JMYUQV6KXEFEFCWMDQP32SMNJA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 15:23:19.051332694 (487h19m17.602803155s) [noise=true , address=145.79.227.24:4583 , piece_id=CFBVKC4JG3QHBY7TCAGS3LLP3VUXV5ZXB5KICXVWV6QOGDNXRRVA, seg_pos={0 0}, upload=429106]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:49260->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:23:19.051374503 (487h19m17.602844974s) [seg_pos={0 0}, upload=429106]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 15:23:47.596917488 (487h19m46.148387949s) [upload=429106]: Done uploading segments... Nov 29 15:23:47.596962294 (487h19m46.148432755s) [upload=429106]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 58.5492323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.......X] duration: 30.267583341s trace: https://jaeger.dev.storj.io/trace/3c12e61589dba0b7 FAIL (2025-11-29 15:23:45 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 34) Duration: 30.267668021s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 15.041998699s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:23:43 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 15.042030931s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X........X.....................] duration: 4.96978668s trace: https://jaeger.dev.storj.io/trace/66b8eb4790a4cf18 FAIL (2025-11-29 15:23:02 UTC) orbiter warning: context deadline exceeded Duration: 4.969814203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.787986521s trace: https://jaeger.dev.storj.io/trace/45a966710995e285 FAIL (2025-11-29 15:22:27 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.788020256s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.980435782s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:22:21 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.980462153s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.782124117s trace: https://jaeger.dev.storj.io/trace/45a966710995e285 FAIL (2025-11-29 15:22:09 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.782143083s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.979055445s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:22:00 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.979082907s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X........X....................] duration: 3.361825736s trace: https://jaeger.dev.storj.io/trace/66b8eb4790a4cf18 FAIL (2025-11-29 15:21:52 UTC) orbiter warning: context deadline exceeded Duration: 3.36185449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.745659774s trace: https://jaeger.dev.storj.io/trace/45a966710995e285 FAIL (2025-11-29 15:21:24 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.745692216s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.036454045s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:21:01 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.036490084s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X..............................X........X...................] duration: 3.457574455s trace: https://jaeger.dev.storj.io/trace/66b8eb4790a4cf18 FAIL (2025-11-29 15:20:55 UTC) orbiter warning: context deadline exceeded Duration: 3.457593893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.0050036s trace: https://jaeger.dev.storj.io/trace/45a966710995e285 FAIL (2025-11-29 15:20:27 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.005033206s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.744912973s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:20:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.744940916s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X..............................X........X..................] duration: 3.496473258s trace: https://jaeger.dev.storj.io/trace/66b8eb4790a4cf18 FAIL (2025-11-29 15:20:08 UTC) orbiter warning: context deadline exceeded Duration: 3.496506672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.777850281s trace: https://jaeger.dev.storj.io/trace/45a966710995e285 FAIL (2025-11-29 15:19:57 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.777913932s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.969001239s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:19:51 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.969019905s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X..............................X........X.................] duration: 3.340614767s trace: https://jaeger.dev.storj.io/trace/66b8eb4790a4cf18 FAIL (2025-11-29 15:19:27 UTC) orbiter warning: context deadline exceeded Duration: 3.340646017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.775312769s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:19:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.775381411s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.064757317s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:19:01 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.064789008s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.974430581s trace: https://jaeger.dev.storj.io/trace/45a966710995e285 FAIL (2025-11-29 15:18:45 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.974499302s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.063312557s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:18:38 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.063364737s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 10.526727277s trace: https://jaeger.dev.storj.io/trace/45a966710995e285 FAIL (2025-11-29 15:18:22 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 10.526743818s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)