Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.94237898s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:46:27 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.942401704s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.......X......X.........................] duration: 6.013794525s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:46:25 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.013824081s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X....XX...........................] duration: 7.320539447s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:45:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 7.320583972s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.049853254s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:45:50 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.049873182s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X........X........................X.........................] duration: 3.451557994s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:45:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.451586469s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.......X......X........................] duration: 5.658887425s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:45:40 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.658919115s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.265794979s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:45:40 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.26581604s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.013424718s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:45:22 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.013448042s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X........X........................X........................] duration: 3.646982249s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:44:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.64701948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X....XX..........................] duration: 5.744154094s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:44:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.744186176s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.......X......X.......................] duration: 5.702183635s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:44:43 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.702216346s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.982364229s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:44:38 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 1.982461656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X....XX.........................] duration: 5.817801395s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:44:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.817835812s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X........X........................X.......................] duration: 3.543436107s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:44:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.543462236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.949006662s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:44:14 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.949034593s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.......X......X......................] duration: 5.752529867s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:44:03 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.75256318s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.986058055s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:43:34 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 1.986086298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......X......X.....................] duration: 5.827038913s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:43:34 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.827102154s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X....XX........................] duration: 5.688086652s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:43:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.688113945s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.971913552s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:43:26 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.971941736s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X........X........................X......................] duration: 3.721011155s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:43:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.721306728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.032213243s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:42:59 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.032245013s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.......X......X....................] duration: 5.580333097s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:42:28 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.580364397s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X....XX.......................] duration: 5.883797871s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:42:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.883830744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.000260868s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:42:24 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.000291948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.996335553s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:42:23 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.996364188s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.994233422s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:42:06 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.994261826s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X........X........................X.....................] duration: 3.740020462s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:42:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.740053935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.014304902s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:41:48 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.014323077s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.......X......X...................] duration: 5.861206026s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:41:37 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.861238998s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.015153548s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:41:34 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.01521804s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X........X........................X....................] duration: 3.447426887s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:41:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447445994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X....XX......................] duration: 5.83067623s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:41:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.830719021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.030807476s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:41:26 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.03083605s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.009476771s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:41:00 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.009537145s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X....XX.....................] duration: 5.810711505s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:40:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.81079822s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X........X........................X...................] duration: 3.364453367s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:40:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.364565582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......X......X..................] duration: 5.700987828s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:40:41 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.701021293s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X....XX....................] duration: 5.764845595s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:40:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.764877849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.961196402s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:40:24 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 1.961224577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......X......X.................] duration: 5.814044297s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:40:23 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.814072772s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.962752846s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:40:22 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.962779316s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X....XX...................] duration: 5.900264761s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:40:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.900286822s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.066260275s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:40:01 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.066290942s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......X........X........................X..................] duration: 3.481299536s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:39:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481357277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.026385144s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:39:57 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 2.02641898s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.......X......X................] duration: 5.888814827s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:39:20 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.888845677s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.01154726s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:39:12 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.011605642s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.975909931s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:38:57 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.97594652s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X....XX..................] duration: 5.882684826s trace: https://jaeger.dev.storj.io/trace/3ab15022a6c35c67 FAIL (2025-11-29 15:38:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:27:13.062783784 (487h23m11.614254235s) [upload=429170]: Starting upload Nov 29 15:27:13.062873785 (487h23m11.614344246s) [upload=429170]: Done starting upload Nov 29 15:27:13.063139451 (487h23m11.614609912s) [upload=429170]: Uploading segments... Nov 29 15:27:13.063191259 (487h23m11.61466172s) [upload=429170]: Got next segment. Inline: false Nov 29 15:27:13.063220445 (487h23m11.614690896s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:27:13.063225775 (487h23m11.614696226s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:27:13.751862653 (487h23m12.303333104s) [seg_pos={0 0}, upload=429170]: Begin upload segment... Nov 29 15:27:13.751920634 (487h23m12.303391085s) [seg_pos={0 0}, upload=429170]: Done begin upload segment. Nov 29 15:27:13.752017006 (487h23m12.303487467s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752098662 (487h23m12.303569154s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.752746175 (487h23m12.304216636s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753097384 (487h23m12.304567845s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753479311 (487h23m12.304949772s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.753953334 (487h23m12.305423805s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754330031 (487h23m12.305800482s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.754697621 (487h23m12.306168072s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.755018462 (487h23m12.306488923s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757170811 (487h23m12.308641302s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757564731 (487h23m12.309035192s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.757953140 (487h23m12.309423591s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.759996171 (487h23m12.311466632s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.760564302 (487h23m12.312034793s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.761244177 (487h23m12.312714628s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:13.762595019 (487h23m12.31406548s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Uploading piece... Nov 29 15:27:15.537297962 (487h23m14.088768423s) [upload=429170]: Next returned nil segment Nov 29 15:27:15.537314273 (487h23m14.088784734s) [upload=429170]: Waiting for error group managing segments... Nov 29 15:27:15.867998906 (487h23m14.419469367s) [noise=true , address=77.90.31.18:4807 , piece_id=RU3FMZJUUPYUHERLD3C6FBXASB4UTTPRNT66NTXRP2KKWEFFJI4Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.886001056 (487h23m14.437471517s) [noise=true , address=77.90.31.12:4590 , piece_id=J4WS2UWSJZ6XYDWYWHOJYBDLKIRDKNWTW2U4AATHBSTD2EFHLXAQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.896643882 (487h23m14.448114343s) [noise=true , address=77.90.31.15:4698 , piece_id=WCPQDZCTAJ42P5DIQG3D3QTCHBD2FXMZQPCF7J7AC7IVBOO2G3PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.932016368 (487h23m14.483486829s) [noise=true , address=45.91.116.8:4045 , piece_id=BEAZ2QNKODHFEFB43MUFHGMA36HFYO44XACP44IQNRAIMFSP7KFA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:15.933181287 (487h23m14.484651758s) [noise=true , address=45.91.116.11:4117 , piece_id=N6O5IDULXY4S72I4J4TPLMI7YTDJOEPEUW7A7JUDOKLUGYV3EJ3Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.785998565 (487h23m18.337469026s) [noise=true , address=145.79.227.7:4295 , piece_id=IBJR7BYU2BWV5LF5IYA3VZUERQTGYDIYF7NEWOPCWGXZ3Q5IOJCA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:19.873379766 (487h23m18.424850217s) [noise=true , address=145.79.227.6:4257 , piece_id=4253WUWLSPJS7WJSG5BLOMYILFBLERYNIQIL774MMXFHY553B4IQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.334605515 (487h23m19.886075976s) [noise=true , address=145.79.227.32:4836 , piece_id=SPPJPZ27D3YGP3FXWNTTHQG4M2ZOJRJONO75VJRGJVJU6N26DTHA, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.349005135 (487h23m19.900475596s) [noise=true , address=145.79.227.26:4655 , piece_id=T6V5LWDRJI7T3YIF27IGPJONGPGK67HMOBIFMKUBF4SPCSVE465A, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.389092420 (487h23m19.940562901s) [noise=true , address=145.79.227.29:4943 , piece_id=YD2WZZDNIWZYJQVUEY4PU7ZXH5J6NN5ZMCYUVNGJMTZ6TTLDLPKQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:21.453406757 (487h23m20.004877229s) [noise=true , address=145.79.227.38:4977 , piece_id=TE5YM2DJDELGYFFWKZGDD5J3IN4KHDXOTDJ4S5LUQYO3Q6SFHXVQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:22.092511197 (487h23m20.643981648s) [noise=true , address=145.79.227.18:4007 , piece_id=WGY2YRIYUYOBWSBF63V7G7OO2V72YUTDR4EITDKWG4SAKVDQDLLQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381386043 (487h23m21.932856504s) [noise=true , address=145.79.227.33:4869 , piece_id=6HCSDFSMHZR2W46SFSBTXR2QELNZAI5K2WFXK5BKBVF3M6RSS5PQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: <nil> Nov 29 15:27:23.381412704 (487h23m21.932883165s) [seg_pos={0 0}, upload=429170]: Segment reached optimal threshold of 13 pieces. Nov 29 15:27:23.382270508 (487h23m21.933740969s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Nov 29 15:27:23.382348887 (487h23m21.933819368s) [noise=true , address=145.79.227.13:4804 , piece_id=AM7XYLVXV7RZSAB7SPLYJVWYI2MEULOENFL72MBLKGJMQPG5FENQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382640041 (487h23m21.934110492s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: read tcp 188.166.225.170:37226->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382650071 (487h23m21.934120522s) [noise=true , address=145.79.227.30:4763 , piece_id=QKALUOQGKT4ZGRDE74GDHQHENGXZ7DORZCCE5X4Q4RP52FKUJ5KQ, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382878906 (487h23m21.934349367s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:54662->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 15:27:23.382889296 (487h23m21.934359757s) [noise=true , address=145.79.227.19:4078 , piece_id=MQHQOQPZRGGUOCUIKDAA7BS7I7NQ2OPOECNMIGLYK72HNHKIVW5Q, seg_pos={0 0}, upload=429170]: Long tail cancellation due to optimal threshold Nov 29 15:27:23.382912710 (487h23m21.934383171s) [seg_pos={0 0}, upload=429170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 29 15:27:23.383075380 (487h23m21.934545851s) [upload=429170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 29 15:27:23.383084537 (487h23m21.934554989s) [upload=429170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:27:26.389944377 (487h23m24.941414838s) [upload=429170]: Done uploading segments... Nov 29 15:27:26.389967832 (487h23m24.941438293s) [upload=429170]: Upload finished. err: <nil> Duration: 5.882713791s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)