Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X....XX.....................] duration: 5.810711505s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:40:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.81079822s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X........X........................X...................] duration: 3.364453367s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:40:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.364565582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......X......X..................] duration: 5.700987828s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:40:41 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.701021293s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X....XX....................] duration: 5.764845595s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:40:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.764877849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.961196402s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:40:24 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 1.961224577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......X......X.................] duration: 5.814044297s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:40:23 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.814072772s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.962752846s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:40:22 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.962779316s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X....XX...................] duration: 5.900264761s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:40:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.900286822s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.066260275s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:40:01 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.066290942s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......X........X........................X..................] duration: 3.481299536s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:39:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481357277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.026385144s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:39:57 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.02641898s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.......X......X................] duration: 5.888814827s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:39:20 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.888845677s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.01154726s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:39:12 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.011605642s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.975909931s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:38:57 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.97594652s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X....XX..................] duration: 5.882684826s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:38:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.882713791s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........X........X........................X.................] duration: 3.537123787s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:38:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537200895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.016286115s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:38:45 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.016314159s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........X........X........................X................] duration: 3.466304034s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:38:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.466329312s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.......X......X...............] duration: 6.001655277s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:38:15 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.001687939s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X....XX.................] duration: 5.832910204s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:38:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.832936896s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.00817593s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:38:05 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.008199104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.945777352s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:37:51 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.94580685s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........X........X........................X...............] duration: 3.640132127s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:37:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.64015439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X....XX................] duration: 5.948132131s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:37:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.948186866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.......X......X..............] duration: 5.938320668s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:37:35 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.938346938s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X....XX...............] duration: 5.790549273s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:37:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.790581324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.07673005s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:37:18 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.076891929s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.201067744s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:37:11 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.201098963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.......X......X.............] duration: 6.004725386s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:37:00 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.004758398s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........X........X........................X..............] duration: 3.495147539s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:36:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.495177917s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.075387745s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:36:51 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.07541132s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.989822623s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:36:34 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.989848713s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.......X......X............] duration: 5.845053561s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:36:31 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.845081384s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X....XX..............] duration: 5.806935869s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:36:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.806979061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.995591031s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:36:18 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 1.995693324s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............X........X........................X.............] duration: 3.555708106s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:36:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555731921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.......X......X...........] duration: 5.724203183s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:35:56 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.724234123s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X........X........................X............] duration: 3.441559822s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:35:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.441588958s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X....XX.............] duration: 5.874386062s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:35:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.874415919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.101482906s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:35:28 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.101499718s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.105952596s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:35:27 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.105979408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.......X......X..........] duration: 5.992457266s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:35:10 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.992488506s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X....XX............] duration: 6.011363557s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:35:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 6.011387351s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X........X........................X...........] duration: 3.343679401s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:35:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.343701964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.043590738s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:34:49 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.043682102s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.078502556s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:34:41 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.078521713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X....XX...........] duration: 5.824755349s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:34:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.824872212s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.......X......X.........] duration: 5.873606707s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:34:36 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.873638869s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X........X........................X..........] duration: 3.689491797s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:34:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.689523528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.03918086s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:34:26 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.039212611s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)