Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.................................X.....X.] duration: 5.819809833s trace: https://jaeger.dev.storj.io/trace/7a18867a8cd817a3 FAIL (2025-12-09 13:22:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.819831755s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X................X..............................X.....X.....] duration: 11.927848047s trace: https://jaeger.dev.storj.io/trace/60a7cbb2e1d6c052 FAIL (2025-12-09 13:22:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.927875331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XXXX..................................X..................] duration: 9.965190361s trace: https://jaeger.dev.storj.io/trace/314592c187a50ec FAIL (2025-12-09 13:22:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, Piece ID: MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ) context deadline exceeded; piecestore: (Node ID: 1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, Piece ID: G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ) context deadline exceeded ======LOGS====== Dec 9 13:08:34.623724641 (93h39m34.494807697s) [upload=81665]: Starting upload Dec 9 13:08:34.623815704 (93h39m34.494898749s) [upload=81665]: Done starting upload Dec 9 13:08:34.623966372 (93h39m34.495049417s) [upload=81665]: Uploading segments... Dec 9 13:08:34.624323702 (93h39m34.495406747s) [upload=81665]: Got next segment. Inline: false Dec 9 13:08:34.624371984 (93h39m34.495455029s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 13:08:34.624377203 (93h39m34.495460238s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 13:08:37.281127755 (93h39m37.15221081s) [seg_pos={0 0}, upload=81665]: Begin upload segment... Dec 9 13:08:37.281189272 (93h39m37.152272317s) [seg_pos={0 0}, upload=81665]: Done begin upload segment. Dec 9 13:08:37.281217316 (93h39m37.152300351s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.281227836 (93h39m37.152310881s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282226888 (93h39m37.153309923s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282666044 (93h39m37.153749079s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283051929 (93h39m37.154134964s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283463072 (93h39m37.154546117s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283879755 (93h39m37.1549628s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.284332708 (93h39m37.155415753s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285609359 (93h39m37.156692404s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285915071 (93h39m37.156998116s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.286766462 (93h39m37.157849497s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287035495 (93h39m37.15811853s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287925931 (93h39m37.159008976s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288456982 (93h39m37.159540027s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288723048 (93h39m37.159806093s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.290630710 (93h39m37.161713775s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:39.186350861 (93h39m39.057433906s) [upload=81665]: Next returned nil segment Dec 9 13:08:39.186369836 (93h39m39.057452871s) [upload=81665]: Waiting for error group managing segments... Dec 9 13:08:39.492788692 (93h39m39.363871737s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.380040727 (93h39m40.251123782s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.382027781 (93h39m40.253110826s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.388157955 (93h39m40.259241s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.403198054 (93h39m40.274281099s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.408072455 (93h39m40.27915549s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.416722717 (93h39m40.287805752s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.419255590 (93h39m40.290338625s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.434848462 (93h39m40.305931507s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.463333804 (93h39m40.334416839s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.467513042 (93h39m40.338596087s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.471574776 (93h39m40.342657822s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586586735 (93h39m40.45766978s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586615669 (93h39m40.457698714s) [seg_pos={0 0}, upload=81665]: Segment reached optimal threshold of 13 pieces. Dec 9 13:08:40.587192418 (93h39m40.458275463s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 188.166.225.170:41030->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587212296 (93h39m40.458295341s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587265738 (93h39m40.458348793s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:33846->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587278271 (93h39m40.458361316s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587358154 (93h39m40.458441199s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: read tcp 188.166.225.170:35084->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587366700 (93h39m40.458449745s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587391287 (93h39m40.458474322s) [seg_pos={0 0}, upload=81665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 13:08:40.587514591 (93h39m40.458597626s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:08:40.587521073 (93h39m40.458604119s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:08:55.391439615 (93h39m55.26252267s) [upload=81665]: Done uploading segments... Dec 9 13:08:55.391462578 (93h39m55.262545613s) [upload=81665]: Upload finished. err: <nil> Duration: 9.965217313s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.................................X.....X] duration: 6m11.644620674s trace: https://jaeger.dev.storj.io/trace/7a18867a8cd817a3 FAIL (2025-12-09 13:22:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6m11.64463986s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 8.380357855s trace: https://jaeger.dev.storj.io/trace/54317c1753c24c83 FAIL (2025-12-09 13:22:17 UTC) orbiter warning: context deadline exceeded Duration: 8.380394223s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..........X............X.......XX.XX...X......X.XX........X.] duration: 5.001278435s trace: https://jaeger.dev.storj.io/trace/5e0e55455f99dc31 FAIL (2025-12-09 13:22:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001302922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.46829538s trace: https://jaeger.dev.storj.io/trace/77ca7d69fd2862c6 FAIL (2025-12-09 13:22:13 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.468326719s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.869437818s trace: https://jaeger.dev.storj.io/trace/435b3249ab0873c2 FAIL (2025-12-09 13:22:12 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.869474587s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 26.691993341s trace: https://jaeger.dev.storj.io/trace/54317c1753c24c83 FAIL (2025-12-09 13:22:01 UTC) orbiter warning: context deadline exceeded Duration: 26.692033366s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...........X............X.......XX.XX...X......X.XX........X] duration: 5.001222262s trace: https://jaeger.dev.storj.io/trace/5e0e55455f99dc31 FAIL (2025-12-09 13:21:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001243523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.452749639s trace: https://jaeger.dev.storj.io/trace/77ca7d69fd2862c6 FAIL (2025-12-09 13:21:48 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.452781929s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 8.233866157s trace: https://jaeger.dev.storj.io/trace/9e51647c84d1fb5 FAIL (2025-12-09 13:21:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469453596 (93h37m4.340536641s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469560099 (93h37m4.340643144s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470052186 (93h37m4.341135231s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470103654 (93h37m4.341186699s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470503986 (93h37m4.341587031s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470603416 (93h37m4.341686471s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471108767 (93h37m4.342191832s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471161488 (93h37m4.342244533s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471570507 (93h37m4.342653562s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471658214 (93h37m4.342741259s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472164348 (93h37m4.343247393s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472240212 (93h37m4.343323257s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472826509 (93h37m4.343909554s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472993457 (93h37m4.344076502s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:06.014093105 (93h37m5.88517615s) [upload=81626]: Next returned nil segment Dec 9 13:06:06.014111329 (93h37m5.885194374s) [upload=81626]: Waiting for error group managing segments... Dec 9 13:06:06.452227624 (93h37m6.323310659s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.477649473 (93h37m6.348732518s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.484904859 (93h37m6.355987894s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.488510093 (93h37m6.359593138s) [noise=true , address=77.90.31.18:4820 , piece_id=SUANM4P45VAKRA7RZRCXOKAU5MP5PY2CCSUEIJ3ECV4Y5NCRXCRQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.491439753 (93h37m6.362522798s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.498682043 (93h37m6.369765078s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.508793167 (93h37m6.379876202s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.527211489 (93h37m6.398294535s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.545142274 (93h37m6.416225319s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.585701106 (93h37m6.456784151s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.181237322 (93h37m7.052320368s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.249731199 (93h37m7.120814244s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.447190704 (93h37m7.318273749s) [noise=true , address=145.79.227.31:4809 , piece_id=NGGDCXTIWQ4UWG2XYZ76PW2CGWONS63WOAMAE5PU5BZVUWYU6HTQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905270685 (93h37m13.77635372s) [noise=true , address=45.91.116.8:4071 , piece_id=4PVVWGWIDNVHIYHZNNSXECZ6VO4VHGQRSPJGOSAEEJ3WXNYQF6NQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905292547 (93h37m13.776375592s) [seg_pos={0 0}, upload=81626]: Segment reached optimal threshold of 14 pieces. Dec 9 13:06:13.905521141 (93h37m13.776604186s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E): protocol: expected piece hash; context canceled; EOF Dec 9 13:06:13.905530960 (93h37m13.776613995s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905599190 (93h37m13.776682225s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection Dec 9 13:06:13.905618116 (93h37m13.776701161s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905716824 (93h37m13.776799869s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection Dec 9 13:06:13.905727003 (93h37m13.776810038s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905954988 (93h37m13.777038043s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W3qDq8BssS8V6osW9NwFHZHMJcCRfjYP8qKwphE5SenVewWXy): protocol: expected piece hash; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; EOF Dec 9 13:06:13.905964836 (93h37m13.777047882s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.906061531 (93h37m13.777144576s) [seg_pos={0 0}, upload=81626]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 13:06:13.906185627 (93h37m13.777268672s) [upload=81626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:06:13.906191748 (93h37m13.777274793s) [upload=81626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:07:02.540166267 (93h38m2.411249312s) [upload=81626]: Done uploading segments... Dec 9 13:07:02.540284373 (93h38m2.411367428s) [upload=81626]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.233890032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.785008029s trace: https://jaeger.dev.storj.io/trace/435b3249ab0873c2 FAIL (2025-12-09 13:21:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.785032306s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XXXX..................................X.................] duration: 16.185476159s trace: https://jaeger.dev.storj.io/trace/314592c187a50ec FAIL (2025-12-09 13:21:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, Piece ID: MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ) context deadline exceeded; piecestore: (Node ID: 1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, Piece ID: G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ) context deadline exceeded ======LOGS====== Dec 9 13:08:34.623724641 (93h39m34.494807697s) [upload=81665]: Starting upload Dec 9 13:08:34.623815704 (93h39m34.494898749s) [upload=81665]: Done starting upload Dec 9 13:08:34.623966372 (93h39m34.495049417s) [upload=81665]: Uploading segments... Dec 9 13:08:34.624323702 (93h39m34.495406747s) [upload=81665]: Got next segment. Inline: false Dec 9 13:08:34.624371984 (93h39m34.495455029s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 13:08:34.624377203 (93h39m34.495460238s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 13:08:37.281127755 (93h39m37.15221081s) [seg_pos={0 0}, upload=81665]: Begin upload segment... Dec 9 13:08:37.281189272 (93h39m37.152272317s) [seg_pos={0 0}, upload=81665]: Done begin upload segment. Dec 9 13:08:37.281217316 (93h39m37.152300351s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.281227836 (93h39m37.152310881s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282226888 (93h39m37.153309923s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282666044 (93h39m37.153749079s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283051929 (93h39m37.154134964s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283463072 (93h39m37.154546117s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283879755 (93h39m37.1549628s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.284332708 (93h39m37.155415753s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285609359 (93h39m37.156692404s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285915071 (93h39m37.156998116s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.286766462 (93h39m37.157849497s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287035495 (93h39m37.15811853s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287925931 (93h39m37.159008976s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288456982 (93h39m37.159540027s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288723048 (93h39m37.159806093s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.290630710 (93h39m37.161713775s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:39.186350861 (93h39m39.057433906s) [upload=81665]: Next returned nil segment Dec 9 13:08:39.186369836 (93h39m39.057452871s) [upload=81665]: Waiting for error group managing segments... Dec 9 13:08:39.492788692 (93h39m39.363871737s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.380040727 (93h39m40.251123782s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.382027781 (93h39m40.253110826s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.388157955 (93h39m40.259241s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.403198054 (93h39m40.274281099s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.408072455 (93h39m40.27915549s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.416722717 (93h39m40.287805752s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.419255590 (93h39m40.290338625s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.434848462 (93h39m40.305931507s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.463333804 (93h39m40.334416839s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.467513042 (93h39m40.338596087s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.471574776 (93h39m40.342657822s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586586735 (93h39m40.45766978s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586615669 (93h39m40.457698714s) [seg_pos={0 0}, upload=81665]: Segment reached optimal threshold of 13 pieces. Dec 9 13:08:40.587192418 (93h39m40.458275463s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 188.166.225.170:41030->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587212296 (93h39m40.458295341s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587265738 (93h39m40.458348793s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:33846->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587278271 (93h39m40.458361316s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587358154 (93h39m40.458441199s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: read tcp 188.166.225.170:35084->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587366700 (93h39m40.458449745s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587391287 (93h39m40.458474322s) [seg_pos={0 0}, upload=81665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 13:08:40.587514591 (93h39m40.458597626s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:08:40.587521073 (93h39m40.458604119s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:08:55.391439615 (93h39m55.26252267s) [upload=81665]: Done uploading segments... Dec 9 13:08:55.391462578 (93h39m55.262545613s) [upload=81665]: Upload finished. err: <nil> Duration: 16.18550345s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [XX................X..............................X.....X....] duration: 12.084617066s trace: https://jaeger.dev.storj.io/trace/60a7cbb2e1d6c052 FAIL (2025-12-09 13:21:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.084665367s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 19.27993546s trace: https://jaeger.dev.storj.io/trace/9e51647c84d1fb5 FAIL (2025-12-09 13:21:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469453596 (93h37m4.340536641s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469560099 (93h37m4.340643144s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470052186 (93h37m4.341135231s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470103654 (93h37m4.341186699s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470503986 (93h37m4.341587031s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470603416 (93h37m4.341686471s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471108767 (93h37m4.342191832s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471161488 (93h37m4.342244533s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471570507 (93h37m4.342653562s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471658214 (93h37m4.342741259s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472164348 (93h37m4.343247393s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472240212 (93h37m4.343323257s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472826509 (93h37m4.343909554s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472993457 (93h37m4.344076502s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:06.014093105 (93h37m5.88517615s) [upload=81626]: Next returned nil segment Dec 9 13:06:06.014111329 (93h37m5.885194374s) [upload=81626]: Waiting for error group managing segments... Dec 9 13:06:06.452227624 (93h37m6.323310659s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.477649473 (93h37m6.348732518s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.484904859 (93h37m6.355987894s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.488510093 (93h37m6.359593138s) [noise=true , address=77.90.31.18:4820 , piece_id=SUANM4P45VAKRA7RZRCXOKAU5MP5PY2CCSUEIJ3ECV4Y5NCRXCRQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.491439753 (93h37m6.362522798s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.498682043 (93h37m6.369765078s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.508793167 (93h37m6.379876202s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.527211489 (93h37m6.398294535s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.545142274 (93h37m6.416225319s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.585701106 (93h37m6.456784151s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.181237322 (93h37m7.052320368s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.249731199 (93h37m7.120814244s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.447190704 (93h37m7.318273749s) [noise=true , address=145.79.227.31:4809 , piece_id=NGGDCXTIWQ4UWG2XYZ76PW2CGWONS63WOAMAE5PU5BZVUWYU6HTQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905270685 (93h37m13.77635372s) [noise=true , address=45.91.116.8:4071 , piece_id=4PVVWGWIDNVHIYHZNNSXECZ6VO4VHGQRSPJGOSAEEJ3WXNYQF6NQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905292547 (93h37m13.776375592s) [seg_pos={0 0}, upload=81626]: Segment reached optimal threshold of 14 pieces. Dec 9 13:06:13.905521141 (93h37m13.776604186s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E): protocol: expected piece hash; context canceled; EOF Dec 9 13:06:13.905530960 (93h37m13.776613995s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905599190 (93h37m13.776682225s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection Dec 9 13:06:13.905618116 (93h37m13.776701161s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905716824 (93h37m13.776799869s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection Dec 9 13:06:13.905727003 (93h37m13.776810038s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905954988 (93h37m13.777038043s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W3qDq8BssS8V6osW9NwFHZHMJcCRfjYP8qKwphE5SenVewWXy): protocol: expected piece hash; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; EOF Dec 9 13:06:13.905964836 (93h37m13.777047882s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.906061531 (93h37m13.777144576s) [seg_pos={0 0}, upload=81626]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 13:06:13.906185627 (93h37m13.777268672s) [upload=81626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:06:13.906191748 (93h37m13.777274793s) [upload=81626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:07:02.540166267 (93h38m2.411249312s) [upload=81626]: Done uploading segments... Dec 9 13:07:02.540284373 (93h38m2.411367428s) [upload=81626]: Upload finished. err: metaclient: context deadline exceeded Duration: 19.27996751s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [............X............X.......XX.XX...X......X.XX........] duration: 4.62060126s trace: https://jaeger.dev.storj.io/trace/496e4bd5466bcf07 FAIL (2025-12-09 13:21:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.620629814s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 30.001778219s trace: https://jaeger.dev.storj.io/trace/54317c1753c24c83 FAIL (2025-12-09 13:20:58 UTC) orbiter warning: context deadline exceeded Duration: 30.001802435s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.435776537s trace: https://jaeger.dev.storj.io/trace/77ca7d69fd2862c6 FAIL (2025-12-09 13:20:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.43580875s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 28.883035932s trace: https://jaeger.dev.storj.io/trace/9e51647c84d1fb5 FAIL (2025-12-09 13:20:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469453596 (93h37m4.340536641s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469560099 (93h37m4.340643144s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470052186 (93h37m4.341135231s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470103654 (93h37m4.341186699s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470503986 (93h37m4.341587031s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470603416 (93h37m4.341686471s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471108767 (93h37m4.342191832s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471161488 (93h37m4.342244533s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471570507 (93h37m4.342653562s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471658214 (93h37m4.342741259s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472164348 (93h37m4.343247393s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472240212 (93h37m4.343323257s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472826509 (93h37m4.343909554s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472993457 (93h37m4.344076502s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:06.014093105 (93h37m5.88517615s) [upload=81626]: Next returned nil segment Dec 9 13:06:06.014111329 (93h37m5.885194374s) [upload=81626]: Waiting for error group managing segments... Dec 9 13:06:06.452227624 (93h37m6.323310659s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.477649473 (93h37m6.348732518s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.484904859 (93h37m6.355987894s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.488510093 (93h37m6.359593138s) [noise=true , address=77.90.31.18:4820 , piece_id=SUANM4P45VAKRA7RZRCXOKAU5MP5PY2CCSUEIJ3ECV4Y5NCRXCRQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.491439753 (93h37m6.362522798s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.498682043 (93h37m6.369765078s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.508793167 (93h37m6.379876202s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.527211489 (93h37m6.398294535s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.545142274 (93h37m6.416225319s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.585701106 (93h37m6.456784151s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.181237322 (93h37m7.052320368s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.249731199 (93h37m7.120814244s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.447190704 (93h37m7.318273749s) [noise=true , address=145.79.227.31:4809 , piece_id=NGGDCXTIWQ4UWG2XYZ76PW2CGWONS63WOAMAE5PU5BZVUWYU6HTQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905270685 (93h37m13.77635372s) [noise=true , address=45.91.116.8:4071 , piece_id=4PVVWGWIDNVHIYHZNNSXECZ6VO4VHGQRSPJGOSAEEJ3WXNYQF6NQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905292547 (93h37m13.776375592s) [seg_pos={0 0}, upload=81626]: Segment reached optimal threshold of 14 pieces. Dec 9 13:06:13.905521141 (93h37m13.776604186s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E): protocol: expected piece hash; context canceled; EOF Dec 9 13:06:13.905530960 (93h37m13.776613995s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905599190 (93h37m13.776682225s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection Dec 9 13:06:13.905618116 (93h37m13.776701161s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905716824 (93h37m13.776799869s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection Dec 9 13:06:13.905727003 (93h37m13.776810038s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905954988 (93h37m13.777038043s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W3qDq8BssS8V6osW9NwFHZHMJcCRfjYP8qKwphE5SenVewWXy): protocol: expected piece hash; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; EOF Dec 9 13:06:13.905964836 (93h37m13.777047882s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.906061531 (93h37m13.777144576s) [seg_pos={0 0}, upload=81626]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 13:06:13.906185627 (93h37m13.777268672s) [upload=81626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:06:13.906191748 (93h37m13.777274793s) [upload=81626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:07:02.540166267 (93h38m2.411249312s) [upload=81626]: Done uploading segments... Dec 9 13:07:02.540284373 (93h38m2.411367428s) [upload=81626]: Upload finished. err: metaclient: context deadline exceeded Duration: 28.883069936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XXXX..................................X................] duration: 14.120421525s trace: https://jaeger.dev.storj.io/trace/314592c187a50ec FAIL (2025-12-09 13:20:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, Piece ID: MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ) context deadline exceeded; piecestore: (Node ID: 1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, Piece ID: G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ) context deadline exceeded ======LOGS====== Dec 9 13:08:34.623724641 (93h39m34.494807697s) [upload=81665]: Starting upload Dec 9 13:08:34.623815704 (93h39m34.494898749s) [upload=81665]: Done starting upload Dec 9 13:08:34.623966372 (93h39m34.495049417s) [upload=81665]: Uploading segments... Dec 9 13:08:34.624323702 (93h39m34.495406747s) [upload=81665]: Got next segment. Inline: false Dec 9 13:08:34.624371984 (93h39m34.495455029s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 13:08:34.624377203 (93h39m34.495460238s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 13:08:37.281127755 (93h39m37.15221081s) [seg_pos={0 0}, upload=81665]: Begin upload segment... Dec 9 13:08:37.281189272 (93h39m37.152272317s) [seg_pos={0 0}, upload=81665]: Done begin upload segment. Dec 9 13:08:37.281217316 (93h39m37.152300351s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.281227836 (93h39m37.152310881s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282226888 (93h39m37.153309923s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282666044 (93h39m37.153749079s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283051929 (93h39m37.154134964s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283463072 (93h39m37.154546117s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283879755 (93h39m37.1549628s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.284332708 (93h39m37.155415753s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285609359 (93h39m37.156692404s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285915071 (93h39m37.156998116s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.286766462 (93h39m37.157849497s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287035495 (93h39m37.15811853s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287925931 (93h39m37.159008976s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288456982 (93h39m37.159540027s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288723048 (93h39m37.159806093s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.290630710 (93h39m37.161713775s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:39.186350861 (93h39m39.057433906s) [upload=81665]: Next returned nil segment Dec 9 13:08:39.186369836 (93h39m39.057452871s) [upload=81665]: Waiting for error group managing segments... Dec 9 13:08:39.492788692 (93h39m39.363871737s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.380040727 (93h39m40.251123782s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.382027781 (93h39m40.253110826s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.388157955 (93h39m40.259241s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.403198054 (93h39m40.274281099s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.408072455 (93h39m40.27915549s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.416722717 (93h39m40.287805752s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.419255590 (93h39m40.290338625s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.434848462 (93h39m40.305931507s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.463333804 (93h39m40.334416839s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.467513042 (93h39m40.338596087s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.471574776 (93h39m40.342657822s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586586735 (93h39m40.45766978s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586615669 (93h39m40.457698714s) [seg_pos={0 0}, upload=81665]: Segment reached optimal threshold of 13 pieces. Dec 9 13:08:40.587192418 (93h39m40.458275463s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 188.166.225.170:41030->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587212296 (93h39m40.458295341s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587265738 (93h39m40.458348793s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:33846->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587278271 (93h39m40.458361316s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587358154 (93h39m40.458441199s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: read tcp 188.166.225.170:35084->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587366700 (93h39m40.458449745s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587391287 (93h39m40.458474322s) [seg_pos={0 0}, upload=81665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 13:08:40.587514591 (93h39m40.458597626s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:08:40.587521073 (93h39m40.458604119s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:08:55.391439615 (93h39m55.26252267s) [upload=81665]: Done uploading segments... Dec 9 13:08:55.391462578 (93h39m55.262545613s) [upload=81665]: Upload finished. err: <nil> Duration: 14.120453254s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.533520887s trace: https://jaeger.dev.storj.io/trace/77ca7d69fd2862c6 FAIL (2025-12-09 13:20:36 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.53355918s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 8.260422178s trace: https://jaeger.dev.storj.io/trace/435b3249ab0873c2 FAIL (2025-12-09 13:20:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 8.260443368s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 21.530092843s trace: https://jaeger.dev.storj.io/trace/9e51647c84d1fb5 FAIL (2025-12-09 13:20:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469453596 (93h37m4.340536641s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469560099 (93h37m4.340643144s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470052186 (93h37m4.341135231s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470103654 (93h37m4.341186699s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470503986 (93h37m4.341587031s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470603416 (93h37m4.341686471s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471108767 (93h37m4.342191832s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471161488 (93h37m4.342244533s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471570507 (93h37m4.342653562s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471658214 (93h37m4.342741259s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472164348 (93h37m4.343247393s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472240212 (93h37m4.343323257s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472826509 (93h37m4.343909554s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472993457 (93h37m4.344076502s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:06.014093105 (93h37m5.88517615s) [upload=81626]: Next returned nil segment Dec 9 13:06:06.014111329 (93h37m5.885194374s) [upload=81626]: Waiting for error group managing segments... Dec 9 13:06:06.452227624 (93h37m6.323310659s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.477649473 (93h37m6.348732518s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.484904859 (93h37m6.355987894s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.488510093 (93h37m6.359593138s) [noise=true , address=77.90.31.18:4820 , piece_id=SUANM4P45VAKRA7RZRCXOKAU5MP5PY2CCSUEIJ3ECV4Y5NCRXCRQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.491439753 (93h37m6.362522798s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.498682043 (93h37m6.369765078s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.508793167 (93h37m6.379876202s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.527211489 (93h37m6.398294535s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.545142274 (93h37m6.416225319s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.585701106 (93h37m6.456784151s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.181237322 (93h37m7.052320368s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.249731199 (93h37m7.120814244s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.447190704 (93h37m7.318273749s) [noise=true , address=145.79.227.31:4809 , piece_id=NGGDCXTIWQ4UWG2XYZ76PW2CGWONS63WOAMAE5PU5BZVUWYU6HTQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905270685 (93h37m13.77635372s) [noise=true , address=45.91.116.8:4071 , piece_id=4PVVWGWIDNVHIYHZNNSXECZ6VO4VHGQRSPJGOSAEEJ3WXNYQF6NQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905292547 (93h37m13.776375592s) [seg_pos={0 0}, upload=81626]: Segment reached optimal threshold of 14 pieces. Dec 9 13:06:13.905521141 (93h37m13.776604186s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E): protocol: expected piece hash; context canceled; EOF Dec 9 13:06:13.905530960 (93h37m13.776613995s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905599190 (93h37m13.776682225s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection Dec 9 13:06:13.905618116 (93h37m13.776701161s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905716824 (93h37m13.776799869s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection Dec 9 13:06:13.905727003 (93h37m13.776810038s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905954988 (93h37m13.777038043s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W3qDq8BssS8V6osW9NwFHZHMJcCRfjYP8qKwphE5SenVewWXy): protocol: expected piece hash; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; EOF Dec 9 13:06:13.905964836 (93h37m13.777047882s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.906061531 (93h37m13.777144576s) [seg_pos={0 0}, upload=81626]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 13:06:13.906185627 (93h37m13.777268672s) [upload=81626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:06:13.906191748 (93h37m13.777274793s) [upload=81626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:07:02.540166267 (93h38m2.411249312s) [upload=81626]: Done uploading segments... Dec 9 13:07:02.540284373 (93h38m2.411367428s) [upload=81626]: Upload finished. err: metaclient: context deadline exceeded Duration: 21.530118351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XXXX..................................X...............] duration: 8.994615847s trace: https://jaeger.dev.storj.io/trace/314592c187a50ec FAIL (2025-12-09 13:20:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, Piece ID: MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ) context deadline exceeded; piecestore: (Node ID: 1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, Piece ID: G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ) context deadline exceeded ======LOGS====== Dec 9 13:08:34.623724641 (93h39m34.494807697s) [upload=81665]: Starting upload Dec 9 13:08:34.623815704 (93h39m34.494898749s) [upload=81665]: Done starting upload Dec 9 13:08:34.623966372 (93h39m34.495049417s) [upload=81665]: Uploading segments... Dec 9 13:08:34.624323702 (93h39m34.495406747s) [upload=81665]: Got next segment. Inline: false Dec 9 13:08:34.624371984 (93h39m34.495455029s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 13:08:34.624377203 (93h39m34.495460238s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 13:08:37.281127755 (93h39m37.15221081s) [seg_pos={0 0}, upload=81665]: Begin upload segment... Dec 9 13:08:37.281189272 (93h39m37.152272317s) [seg_pos={0 0}, upload=81665]: Done begin upload segment. Dec 9 13:08:37.281217316 (93h39m37.152300351s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.281227836 (93h39m37.152310881s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282226888 (93h39m37.153309923s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282666044 (93h39m37.153749079s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283051929 (93h39m37.154134964s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283463072 (93h39m37.154546117s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283879755 (93h39m37.1549628s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.284332708 (93h39m37.155415753s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285609359 (93h39m37.156692404s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285915071 (93h39m37.156998116s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.286766462 (93h39m37.157849497s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287035495 (93h39m37.15811853s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287925931 (93h39m37.159008976s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288456982 (93h39m37.159540027s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288723048 (93h39m37.159806093s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.290630710 (93h39m37.161713775s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:39.186350861 (93h39m39.057433906s) [upload=81665]: Next returned nil segment Dec 9 13:08:39.186369836 (93h39m39.057452871s) [upload=81665]: Waiting for error group managing segments... Dec 9 13:08:39.492788692 (93h39m39.363871737s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.380040727 (93h39m40.251123782s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.382027781 (93h39m40.253110826s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.388157955 (93h39m40.259241s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.403198054 (93h39m40.274281099s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.408072455 (93h39m40.27915549s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.416722717 (93h39m40.287805752s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.419255590 (93h39m40.290338625s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.434848462 (93h39m40.305931507s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.463333804 (93h39m40.334416839s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.467513042 (93h39m40.338596087s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.471574776 (93h39m40.342657822s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586586735 (93h39m40.45766978s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586615669 (93h39m40.457698714s) [seg_pos={0 0}, upload=81665]: Segment reached optimal threshold of 13 pieces. Dec 9 13:08:40.587192418 (93h39m40.458275463s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 188.166.225.170:41030->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587212296 (93h39m40.458295341s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587265738 (93h39m40.458348793s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:33846->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587278271 (93h39m40.458361316s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587358154 (93h39m40.458441199s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: read tcp 188.166.225.170:35084->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587366700 (93h39m40.458449745s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587391287 (93h39m40.458474322s) [seg_pos={0 0}, upload=81665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 13:08:40.587514591 (93h39m40.458597626s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:08:40.587521073 (93h39m40.458604119s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:08:55.391439615 (93h39m55.26252267s) [upload=81665]: Done uploading segments... Dec 9 13:08:55.391462578 (93h39m55.262545613s) [upload=81665]: Upload finished. err: <nil> Duration: 8.994704696s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.............X............X.......XX.XX...X......X.XX.......] duration: 5.001098572s trace: https://jaeger.dev.storj.io/trace/496e4bd5466bcf07 FAIL (2025-12-09 13:20:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001159568s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [XXX................X..............................X.....X...] duration: 6.467851887s trace: https://jaeger.dev.storj.io/trace/60a7cbb2e1d6c052 FAIL (2025-12-09 13:20:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.467955555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 7.492264136s trace: https://jaeger.dev.storj.io/trace/435b3249ab0873c2 FAIL (2025-12-09 13:20:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.492295245s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.534633512s trace: https://jaeger.dev.storj.io/trace/77ca7d69fd2862c6 FAIL (2025-12-09 13:20:08 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.534681734s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X.............X............X.......XX.XX...X......X.XX......] duration: 3.513541653s trace: https://jaeger.dev.storj.io/trace/496e4bd5466bcf07 FAIL (2025-12-09 13:19:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513591778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 8.263611937s trace: https://jaeger.dev.storj.io/trace/9e51647c84d1fb5 FAIL (2025-12-09 13:19:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469453596 (93h37m4.340536641s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469560099 (93h37m4.340643144s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470052186 (93h37m4.341135231s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470103654 (93h37m4.341186699s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470503986 (93h37m4.341587031s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470603416 (93h37m4.341686471s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471108767 (93h37m4.342191832s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471161488 (93h37m4.342244533s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471570507 (93h37m4.342653562s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471658214 (93h37m4.342741259s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472164348 (93h37m4.343247393s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472240212 (93h37m4.343323257s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472826509 (93h37m4.343909554s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472993457 (93h37m4.344076502s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:06.014093105 (93h37m5.88517615s) [upload=81626]: Next returned nil segment Dec 9 13:06:06.014111329 (93h37m5.885194374s) [upload=81626]: Waiting for error group managing segments... Dec 9 13:06:06.452227624 (93h37m6.323310659s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.477649473 (93h37m6.348732518s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.484904859 (93h37m6.355987894s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.488510093 (93h37m6.359593138s) [noise=true , address=77.90.31.18:4820 , piece_id=SUANM4P45VAKRA7RZRCXOKAU5MP5PY2CCSUEIJ3ECV4Y5NCRXCRQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.491439753 (93h37m6.362522798s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.498682043 (93h37m6.369765078s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.508793167 (93h37m6.379876202s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.527211489 (93h37m6.398294535s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.545142274 (93h37m6.416225319s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.585701106 (93h37m6.456784151s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.181237322 (93h37m7.052320368s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.249731199 (93h37m7.120814244s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.447190704 (93h37m7.318273749s) [noise=true , address=145.79.227.31:4809 , piece_id=NGGDCXTIWQ4UWG2XYZ76PW2CGWONS63WOAMAE5PU5BZVUWYU6HTQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905270685 (93h37m13.77635372s) [noise=true , address=45.91.116.8:4071 , piece_id=4PVVWGWIDNVHIYHZNNSXECZ6VO4VHGQRSPJGOSAEEJ3WXNYQF6NQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905292547 (93h37m13.776375592s) [seg_pos={0 0}, upload=81626]: Segment reached optimal threshold of 14 pieces. Dec 9 13:06:13.905521141 (93h37m13.776604186s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E): protocol: expected piece hash; context canceled; EOF Dec 9 13:06:13.905530960 (93h37m13.776613995s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905599190 (93h37m13.776682225s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection Dec 9 13:06:13.905618116 (93h37m13.776701161s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905716824 (93h37m13.776799869s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection Dec 9 13:06:13.905727003 (93h37m13.776810038s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905954988 (93h37m13.777038043s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W3qDq8BssS8V6osW9NwFHZHMJcCRfjYP8qKwphE5SenVewWXy): protocol: expected piece hash; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; EOF Dec 9 13:06:13.905964836 (93h37m13.777047882s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.906061531 (93h37m13.777144576s) [seg_pos={0 0}, upload=81626]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 13:06:13.906185627 (93h37m13.777268672s) [upload=81626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:06:13.906191748 (93h37m13.777274793s) [upload=81626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:07:02.540166267 (93h38m2.411249312s) [upload=81626]: Done uploading segments... Dec 9 13:07:02.540284373 (93h38m2.411367428s) [upload=81626]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.26364513s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.161152778s trace: https://jaeger.dev.storj.io/trace/54317c1753c24c83 FAIL (2025-12-09 13:19:30 UTC) orbiter warning: context deadline exceeded Duration: 6.161169411s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......XXXX..................................X..............] duration: 8.957356769s trace: https://jaeger.dev.storj.io/trace/314592c187a50ec FAIL (2025-12-09 13:19:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, Piece ID: MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ) context deadline exceeded; piecestore: (Node ID: 1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, Piece ID: G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ) context deadline exceeded ======LOGS====== Dec 9 13:08:34.623724641 (93h39m34.494807697s) [upload=81665]: Starting upload Dec 9 13:08:34.623815704 (93h39m34.494898749s) [upload=81665]: Done starting upload Dec 9 13:08:34.623966372 (93h39m34.495049417s) [upload=81665]: Uploading segments... Dec 9 13:08:34.624323702 (93h39m34.495406747s) [upload=81665]: Got next segment. Inline: false Dec 9 13:08:34.624371984 (93h39m34.495455029s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 13:08:34.624377203 (93h39m34.495460238s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 13:08:37.281127755 (93h39m37.15221081s) [seg_pos={0 0}, upload=81665]: Begin upload segment... Dec 9 13:08:37.281189272 (93h39m37.152272317s) [seg_pos={0 0}, upload=81665]: Done begin upload segment. Dec 9 13:08:37.281217316 (93h39m37.152300351s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.281227836 (93h39m37.152310881s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282226888 (93h39m37.153309923s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282666044 (93h39m37.153749079s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283051929 (93h39m37.154134964s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283463072 (93h39m37.154546117s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283879755 (93h39m37.1549628s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.284332708 (93h39m37.155415753s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285609359 (93h39m37.156692404s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285915071 (93h39m37.156998116s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.286766462 (93h39m37.157849497s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287035495 (93h39m37.15811853s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287925931 (93h39m37.159008976s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288456982 (93h39m37.159540027s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288723048 (93h39m37.159806093s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.290630710 (93h39m37.161713775s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:39.186350861 (93h39m39.057433906s) [upload=81665]: Next returned nil segment Dec 9 13:08:39.186369836 (93h39m39.057452871s) [upload=81665]: Waiting for error group managing segments... Dec 9 13:08:39.492788692 (93h39m39.363871737s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.380040727 (93h39m40.251123782s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.382027781 (93h39m40.253110826s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.388157955 (93h39m40.259241s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.403198054 (93h39m40.274281099s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.408072455 (93h39m40.27915549s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.416722717 (93h39m40.287805752s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.419255590 (93h39m40.290338625s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.434848462 (93h39m40.305931507s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.463333804 (93h39m40.334416839s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.467513042 (93h39m40.338596087s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.471574776 (93h39m40.342657822s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586586735 (93h39m40.45766978s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586615669 (93h39m40.457698714s) [seg_pos={0 0}, upload=81665]: Segment reached optimal threshold of 13 pieces. Dec 9 13:08:40.587192418 (93h39m40.458275463s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 188.166.225.170:41030->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587212296 (93h39m40.458295341s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587265738 (93h39m40.458348793s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:33846->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587278271 (93h39m40.458361316s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587358154 (93h39m40.458441199s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: read tcp 188.166.225.170:35084->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587366700 (93h39m40.458449745s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587391287 (93h39m40.458474322s) [seg_pos={0 0}, upload=81665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 13:08:40.587514591 (93h39m40.458597626s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:08:40.587521073 (93h39m40.458604119s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:08:55.391439615 (93h39m55.26252267s) [upload=81665]: Done uploading segments... Dec 9 13:08:55.391462578 (93h39m55.262545613s) [upload=81665]: Upload finished. err: <nil> Duration: 8.957385985s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 19.891860205s trace: https://jaeger.dev.storj.io/trace/54317c1753c24c83 FAIL (2025-12-09 13:19:24 UTC) orbiter warning: context deadline exceeded Duration: 19.891924748s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 9.468236497s trace: https://jaeger.dev.storj.io/trace/435b3249ab0873c2 FAIL (2025-12-09 13:19:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 9.468269349s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 20.664285722s trace: https://jaeger.dev.storj.io/trace/9e51647c84d1fb5 FAIL (2025-12-09 13:19:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469453596 (93h37m4.340536641s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469560099 (93h37m4.340643144s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470052186 (93h37m4.341135231s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470103654 (93h37m4.341186699s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470503986 (93h37m4.341587031s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470603416 (93h37m4.341686471s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471108767 (93h37m4.342191832s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471161488 (93h37m4.342244533s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471570507 (93h37m4.342653562s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471658214 (93h37m4.342741259s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472164348 (93h37m4.343247393s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472240212 (93h37m4.343323257s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472826509 (93h37m4.343909554s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472993457 (93h37m4.344076502s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:06.014093105 (93h37m5.88517615s) [upload=81626]: Next returned nil segment Dec 9 13:06:06.014111329 (93h37m5.885194374s) [upload=81626]: Waiting for error group managing segments... Dec 9 13:06:06.452227624 (93h37m6.323310659s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.477649473 (93h37m6.348732518s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.484904859 (93h37m6.355987894s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.488510093 (93h37m6.359593138s) [noise=true , address=77.90.31.18:4820 , piece_id=SUANM4P45VAKRA7RZRCXOKAU5MP5PY2CCSUEIJ3ECV4Y5NCRXCRQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.491439753 (93h37m6.362522798s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.498682043 (93h37m6.369765078s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.508793167 (93h37m6.379876202s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.527211489 (93h37m6.398294535s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.545142274 (93h37m6.416225319s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.585701106 (93h37m6.456784151s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.181237322 (93h37m7.052320368s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.249731199 (93h37m7.120814244s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.447190704 (93h37m7.318273749s) [noise=true , address=145.79.227.31:4809 , piece_id=NGGDCXTIWQ4UWG2XYZ76PW2CGWONS63WOAMAE5PU5BZVUWYU6HTQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905270685 (93h37m13.77635372s) [noise=true , address=45.91.116.8:4071 , piece_id=4PVVWGWIDNVHIYHZNNSXECZ6VO4VHGQRSPJGOSAEEJ3WXNYQF6NQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905292547 (93h37m13.776375592s) [seg_pos={0 0}, upload=81626]: Segment reached optimal threshold of 14 pieces. Dec 9 13:06:13.905521141 (93h37m13.776604186s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E): protocol: expected piece hash; context canceled; EOF Dec 9 13:06:13.905530960 (93h37m13.776613995s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905599190 (93h37m13.776682225s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection Dec 9 13:06:13.905618116 (93h37m13.776701161s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905716824 (93h37m13.776799869s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection Dec 9 13:06:13.905727003 (93h37m13.776810038s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905954988 (93h37m13.777038043s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W3qDq8BssS8V6osW9NwFHZHMJcCRfjYP8qKwphE5SenVewWXy): protocol: expected piece hash; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; EOF Dec 9 13:06:13.905964836 (93h37m13.777047882s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.906061531 (93h37m13.777144576s) [seg_pos={0 0}, upload=81626]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 13:06:13.906185627 (93h37m13.777268672s) [upload=81626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:06:13.906191748 (93h37m13.777274793s) [upload=81626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:07:02.540166267 (93h38m2.411249312s) [upload=81626]: Done uploading segments... Dec 9 13:07:02.540284373 (93h38m2.411367428s) [upload=81626]: Upload finished. err: metaclient: context deadline exceeded Duration: 20.664333224s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.XXX................X..............................X.....X..] duration: 13.980370196s trace: https://jaeger.dev.storj.io/trace/60a7cbb2e1d6c052 FAIL (2025-12-09 13:19:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.980452432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.405211868s trace: https://jaeger.dev.storj.io/trace/77ca7d69fd2862c6 FAIL (2025-12-09 13:18:57 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.405232556s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..XXX................X..............................X.....X.] duration: 8.592477997s trace: https://jaeger.dev.storj.io/trace/60a7cbb2e1d6c052 FAIL (2025-12-09 13:18:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.592495931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 7.698155657s trace: https://jaeger.dev.storj.io/trace/435b3249ab0873c2 FAIL (2025-12-09 13:18:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.698186736s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 18.948550764s trace: https://jaeger.dev.storj.io/trace/54317c1753c24c83 FAIL (2025-12-09 13:18:43 UTC) orbiter warning: context deadline exceeded Duration: 18.948581803s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...XXX................X..............................X.....X] duration: 1m0.000947517s trace: https://jaeger.dev.storj.io/trace/60a7cbb2e1d6c052 FAIL (2025-12-09 13:18:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001001019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........XXXX..................................X.............] duration: 9.401822111s trace: https://jaeger.dev.storj.io/trace/314592c187a50ec FAIL (2025-12-09 13:18:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, Piece ID: MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ) context deadline exceeded; piecestore: (Node ID: 1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, Piece ID: G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ) context deadline exceeded ======LOGS====== Dec 9 13:08:34.623724641 (93h39m34.494807697s) [upload=81665]: Starting upload Dec 9 13:08:34.623815704 (93h39m34.494898749s) [upload=81665]: Done starting upload Dec 9 13:08:34.623966372 (93h39m34.495049417s) [upload=81665]: Uploading segments... Dec 9 13:08:34.624323702 (93h39m34.495406747s) [upload=81665]: Got next segment. Inline: false Dec 9 13:08:34.624371984 (93h39m34.495455029s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 13:08:34.624377203 (93h39m34.495460238s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 13:08:37.281127755 (93h39m37.15221081s) [seg_pos={0 0}, upload=81665]: Begin upload segment... Dec 9 13:08:37.281189272 (93h39m37.152272317s) [seg_pos={0 0}, upload=81665]: Done begin upload segment. Dec 9 13:08:37.281217316 (93h39m37.152300351s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.281227836 (93h39m37.152310881s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282226888 (93h39m37.153309923s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282666044 (93h39m37.153749079s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283051929 (93h39m37.154134964s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283463072 (93h39m37.154546117s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283879755 (93h39m37.1549628s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.284332708 (93h39m37.155415753s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285609359 (93h39m37.156692404s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285915071 (93h39m37.156998116s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.286766462 (93h39m37.157849497s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287035495 (93h39m37.15811853s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287925931 (93h39m37.159008976s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288456982 (93h39m37.159540027s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288723048 (93h39m37.159806093s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.290630710 (93h39m37.161713775s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:39.186350861 (93h39m39.057433906s) [upload=81665]: Next returned nil segment Dec 9 13:08:39.186369836 (93h39m39.057452871s) [upload=81665]: Waiting for error group managing segments... Dec 9 13:08:39.492788692 (93h39m39.363871737s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.380040727 (93h39m40.251123782s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.382027781 (93h39m40.253110826s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.388157955 (93h39m40.259241s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.403198054 (93h39m40.274281099s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.408072455 (93h39m40.27915549s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.416722717 (93h39m40.287805752s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.419255590 (93h39m40.290338625s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.434848462 (93h39m40.305931507s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.463333804 (93h39m40.334416839s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.467513042 (93h39m40.338596087s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.471574776 (93h39m40.342657822s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586586735 (93h39m40.45766978s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586615669 (93h39m40.457698714s) [seg_pos={0 0}, upload=81665]: Segment reached optimal threshold of 13 pieces. Dec 9 13:08:40.587192418 (93h39m40.458275463s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 188.166.225.170:41030->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587212296 (93h39m40.458295341s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587265738 (93h39m40.458348793s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:33846->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587278271 (93h39m40.458361316s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587358154 (93h39m40.458441199s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: read tcp 188.166.225.170:35084->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587366700 (93h39m40.458449745s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587391287 (93h39m40.458474322s) [seg_pos={0 0}, upload=81665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 13:08:40.587514591 (93h39m40.458597626s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:08:40.587521073 (93h39m40.458604119s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:08:55.391439615 (93h39m55.26252267s) [upload=81665]: Done uploading segments... Dec 9 13:08:55.391462578 (93h39m55.262545613s) [upload=81665]: Upload finished. err: <nil> Duration: 9.40185849s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.X.............X............X.......XX.XX...X......X.XX.....] duration: 3.574499323s trace: https://jaeger.dev.storj.io/trace/496e4bd5466bcf07 FAIL (2025-12-09 13:18:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.574529641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.403841049s trace: https://jaeger.dev.storj.io/trace/77ca7d69fd2862c6 FAIL (2025-12-09 13:18:11 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.403877449s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 22.771774377s trace: https://jaeger.dev.storj.io/trace/9e51647c84d1fb5 FAIL (2025-12-09 13:18:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469453596 (93h37m4.340536641s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.469560099 (93h37m4.340643144s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470052186 (93h37m4.341135231s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470103654 (93h37m4.341186699s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470503986 (93h37m4.341587031s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.470603416 (93h37m4.341686471s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471108767 (93h37m4.342191832s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471161488 (93h37m4.342244533s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471570507 (93h37m4.342653562s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.471658214 (93h37m4.342741259s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472164348 (93h37m4.343247393s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472240212 (93h37m4.343323257s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472826509 (93h37m4.343909554s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:04.472993457 (93h37m4.344076502s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Uploading piece... Dec 9 13:06:06.014093105 (93h37m5.88517615s) [upload=81626]: Next returned nil segment Dec 9 13:06:06.014111329 (93h37m5.885194374s) [upload=81626]: Waiting for error group managing segments... Dec 9 13:06:06.452227624 (93h37m6.323310659s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.477649473 (93h37m6.348732518s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.484904859 (93h37m6.355987894s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.488510093 (93h37m6.359593138s) [noise=true , address=77.90.31.18:4820 , piece_id=SUANM4P45VAKRA7RZRCXOKAU5MP5PY2CCSUEIJ3ECV4Y5NCRXCRQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.491439753 (93h37m6.362522798s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.498682043 (93h37m6.369765078s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.508793167 (93h37m6.379876202s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.527211489 (93h37m6.398294535s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.545142274 (93h37m6.416225319s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:06.585701106 (93h37m6.456784151s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.181237322 (93h37m7.052320368s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.249731199 (93h37m7.120814244s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:07.447190704 (93h37m7.318273749s) [noise=true , address=145.79.227.31:4809 , piece_id=NGGDCXTIWQ4UWG2XYZ76PW2CGWONS63WOAMAE5PU5BZVUWYU6HTQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905270685 (93h37m13.77635372s) [noise=true , address=45.91.116.8:4071 , piece_id=4PVVWGWIDNVHIYHZNNSXECZ6VO4VHGQRSPJGOSAEEJ3WXNYQF6NQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil> Dec 9 13:06:13.905292547 (93h37m13.776375592s) [seg_pos={0 0}, upload=81626]: Segment reached optimal threshold of 14 pieces. Dec 9 13:06:13.905521141 (93h37m13.776604186s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E): protocol: expected piece hash; context canceled; EOF Dec 9 13:06:13.905530960 (93h37m13.776613995s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905599190 (93h37m13.776682225s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection Dec 9 13:06:13.905618116 (93h37m13.776701161s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905716824 (93h37m13.776799869s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection Dec 9 13:06:13.905727003 (93h37m13.776810038s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.905954988 (93h37m13.777038043s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W3qDq8BssS8V6osW9NwFHZHMJcCRfjYP8qKwphE5SenVewWXy): protocol: expected piece hash; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; EOF Dec 9 13:06:13.905964836 (93h37m13.777047882s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold Dec 9 13:06:13.906061531 (93h37m13.777144576s) [seg_pos={0 0}, upload=81626]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 13:06:13.906185627 (93h37m13.777268672s) [upload=81626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:06:13.906191748 (93h37m13.777274793s) [upload=81626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:07:02.540166267 (93h38m2.411249312s) [upload=81626]: Done uploading segments... Dec 9 13:07:02.540284373 (93h38m2.411367428s) [upload=81626]: Upload finished. err: metaclient: context deadline exceeded Duration: 22.771803181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.44634045s trace: https://jaeger.dev.storj.io/trace/435b3249ab0873c2 FAIL (2025-12-09 13:18:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.446369906s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........XXXX..................................X............] duration: 14.088504909s trace: https://jaeger.dev.storj.io/trace/314592c187a50ec FAIL (2025-12-09 13:17:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, Piece ID: MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ) context deadline exceeded; piecestore: (Node ID: 1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, Piece ID: G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ) context deadline exceeded ======LOGS====== Dec 9 13:08:34.623724641 (93h39m34.494807697s) [upload=81665]: Starting upload Dec 9 13:08:34.623815704 (93h39m34.494898749s) [upload=81665]: Done starting upload Dec 9 13:08:34.623966372 (93h39m34.495049417s) [upload=81665]: Uploading segments... Dec 9 13:08:34.624323702 (93h39m34.495406747s) [upload=81665]: Got next segment. Inline: false Dec 9 13:08:34.624371984 (93h39m34.495455029s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 13:08:34.624377203 (93h39m34.495460238s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 13:08:37.281127755 (93h39m37.15221081s) [seg_pos={0 0}, upload=81665]: Begin upload segment... Dec 9 13:08:37.281189272 (93h39m37.152272317s) [seg_pos={0 0}, upload=81665]: Done begin upload segment. Dec 9 13:08:37.281217316 (93h39m37.152300351s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.281227836 (93h39m37.152310881s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282226888 (93h39m37.153309923s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.282666044 (93h39m37.153749079s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283051929 (93h39m37.154134964s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283463072 (93h39m37.154546117s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.283879755 (93h39m37.1549628s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.284332708 (93h39m37.155415753s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285609359 (93h39m37.156692404s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.285915071 (93h39m37.156998116s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.286766462 (93h39m37.157849497s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287035495 (93h39m37.15811853s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.287925931 (93h39m37.159008976s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288456982 (93h39m37.159540027s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.288723048 (93h39m37.159806093s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:37.290630710 (93h39m37.161713775s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Uploading piece... Dec 9 13:08:39.186350861 (93h39m39.057433906s) [upload=81665]: Next returned nil segment Dec 9 13:08:39.186369836 (93h39m39.057452871s) [upload=81665]: Waiting for error group managing segments... Dec 9 13:08:39.492788692 (93h39m39.363871737s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.380040727 (93h39m40.251123782s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.382027781 (93h39m40.253110826s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.388157955 (93h39m40.259241s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.403198054 (93h39m40.274281099s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.408072455 (93h39m40.27915549s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.416722717 (93h39m40.287805752s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.419255590 (93h39m40.290338625s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.434848462 (93h39m40.305931507s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.463333804 (93h39m40.334416839s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.467513042 (93h39m40.338596087s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.471574776 (93h39m40.342657822s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586586735 (93h39m40.45766978s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil> Dec 9 13:08:40.586615669 (93h39m40.457698714s) [seg_pos={0 0}, upload=81665]: Segment reached optimal threshold of 13 pieces. Dec 9 13:08:40.587192418 (93h39m40.458275463s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 188.166.225.170:41030->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587212296 (93h39m40.458295341s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587265738 (93h39m40.458348793s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:33846->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587278271 (93h39m40.458361316s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587358154 (93h39m40.458441199s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: read tcp 188.166.225.170:35084->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 13:08:40.587366700 (93h39m40.458449745s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold Dec 9 13:08:40.587391287 (93h39m40.458474322s) [seg_pos={0 0}, upload=81665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 13:08:40.587514591 (93h39m40.458597626s) [upload=81665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:08:40.587521073 (93h39m40.458604119s) [upload=81665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:08:55.391439615 (93h39m55.26252267s) [upload=81665]: Done uploading segments... Dec 9 13:08:55.391462578 (93h39m55.262545613s) [upload=81665]: Upload finished. err: <nil> Duration: 14.088535198s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 20.689022806s trace: https://jaeger.dev.storj.io/trace/54317c1753c24c83 FAIL (2025-12-09 13:17:49 UTC) orbiter warning: context deadline exceeded Duration: 20.689435201s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.439401271s trace: https://jaeger.dev.storj.io/trace/77ca7d69fd2862c6 FAIL (2025-12-09 13:17:46 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.439430566s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)